[ 1860.541155] LustreError: lustre-OST0000-osc-ffff98afd0025800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1863.403488] Lustre: DEBUG MARKER: recovery-small test_24b: @@@@@@ IGNORE (bz5494): multiop didn't fail fsync: 5 or close: 0
[ 1866.572301] Lustre: DEBUG MARKER: recovery-small test_24b: @@@@@@ FAIL: no discarded dirty page found!
[ 1884.404833] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 23:33:40 (1788838420)
[ 1886.679776] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1888.583049] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 23:33:44 (1788838424)
[ 1890.167140] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1891.674515] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 23:33:47 (1788838427)
[ 1895.474871] LustreError: lustre-MDT0000-mdc-ffff98afd0025800: operation mds_reint to node 192.168.203.123@tcp failed: rc = -19
[ 1895.484496] LustreError: Skipped 1 previous similar message
[ 1913.311298] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 192.168.203.123@tcp) was lost; in progress operations using this service will fail
[ 1923.521410] Lustre: Evicted from MGS (at 192.168.203.123@tcp) after server handle changed from 0xff98f31736b3b964 to 0xff98f31736b3d3b9
[ 2017.883785] LustreError: lustre-MDT0000-mdc-ffff98afd0025800: operation ldlm_enqueue to node 192.168.203.123@tcp failed: rc = -19
[ 2035.167582] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 192.168.203.123@tcp) was lost; in progress operations using this service will fail
[ 2045.363027] Lustre: Evicted from MGS (at 192.168.203.123@tcp) after server handle changed from 0xff98f31736b3d3b9 to 0xff98f31736b578b5
[ 2061.978662] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 23:36:37 (1788838597)
[ 2062.469165] Lustre: *** cfs_fail_loc=305, val=0***
[ 2062.475320] Lustre: Skipped 2 previous similar messages
[ 2082.317871] LustreError: lustre-OST0000-osc-ffff98afd0025800: operation ost_connect to node 192.168.203.123@tcp failed: rc = -75
[ 2107.871307] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 192.168.203.123@tcp) was lost; in progress operations using this service will fail
[ 2113.008568] Lustre: lustre-MDT0000-mdc-ffff98afd0025800: Connection to lustre-MDT0000 (at 192.168.203.123@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2113.018483] Lustre: Skipped 11 previous similar messages
[ 2113.157801] Lustre: lustre-MDT0000-mdc-ffff98afd0025800: Connection restored to 192.168.203.123@tcp (at 192.168.203.123@tcp)
[ 2113.170162] Lustre: Skipped 13 previous similar messages
[ 2119.084541] Lustre: Evicted from MGS (at 192.168.203.123@tcp) after server handle changed from 0xff98f31736b578b5 to 0xff98f31736b57e88
[ 2123.705450] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2125.081450] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2132.718626] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 23:37:48 (1788838668)
[ 2144.742876] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 192.168.203.123@tcp) was lost; in progress operations using this service will fail
[ 2144.919951] Lustre: Evicted from MGS (at 192.168.203.123@tcp) after server handle changed from 0xff98f31736b57e88 to 0xff98f31736b57ee3
[ 2147.122477] LustreError: lustre-MDT0000-mdc-ffff98afd0025800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2168.400932] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 23:38:24 (1788838704)
[ 2207.442385] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 23:39:03 (1788838743)
[ 2221.020822] LustreError: lustre-MDT0000-mdc-ffff98afd0025800: operation mds_reint to node 192.168.203.123@tcp failed: rc = -19
[ 2237.471182] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 192.168.203.123@tcp) was lost; in progress operations using this service will fail
[ 2246.761176] Lustre: Evicted from MGS (at 192.168.203.123@tcp) after server handle changed from 0xff98f31736b57ee3 to 0xff98f31736b5efb5
[ 2254.960599] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2256.807298] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2340.319126] Lustre: 2351:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788838861/real 1788838861]  req@ffff98afc629b800 x1875731135573376/t0(0) o400->MGC192.168.203.123@tcp@192.168.203.123@tcp:26/25 lens 224/224 e 0 to 1 dl 1788838877 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2340.354614] Lustre: 2351:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 13 previous similar messages
[ 2340.364618] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 192.168.203.123@tcp) was lost; in progress operations using this service will fail
[ 2350.573284] Lustre: Evicted from MGS (at 192.168.203.123@tcp) after server handle changed from 0xff98f31736b5efb5 to 0xff98f31736b797f2
[ 2366.715492] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2368.954358] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2453.983648] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 192.168.203.123@tcp) was lost; in progress operations using this service will fail
[ 2464.198762] Lustre: Evicted from MGS (at 192.168.203.123@tcp) after server handle changed from 0xff98f31736b797f2 to 0xff98f31736b915f0
[ 2481.983585] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2484.991698] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2516.312744] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 23:44:11 (1788839051)
[ 2520.264719] LustreError: lustre-MDT0000-mdc-ffff98afd0025800: operation mds_reint to node 192.168.203.123@tcp failed: rc = -19
[ 2520.269146] LustreError: Skipped 3 previous similar messages
[ 2539.020213] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 192.168.203.123@tcp) was lost; in progress operations using this service will fail
[ 2539.074437] Lustre: Evicted from MGS (at 192.168.203.123@tcp) after server handle changed from 0xff98f31736b915f0 to 0xff98f31736b9cb25
[ 2552.775868] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2595.504485] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2631.997536] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2675.154114] Lustre: Evicted from MGS (at 192.168.203.123@tcp) after server handle changed from 0xff98f31736ba46bd to 0xff98f31736ba6b14
[ 2675.171724] Lustre: Skipped 2 previous similar messages
[ 2684.412813] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2726.449041] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 192.168.203.123@tcp) was lost; in progress operations using this service will fail
[ 2726.466651] LustreError: Skipped 3 previous similar messages
[ 2736.691681] Lustre: MGC192.168.203.123@tcp: Connection restored to 192.168.203.123@tcp (at 192.168.203.123@tcp)
[ 2736.704638] Lustre: Skipped 17 previous similar messages
[ 2744.319714] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2772.726035] Lustre: lustre-MDT0000-mdc-ffff98afd0025800: Connection to lustre-MDT0000 (at 192.168.203.123@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2772.745910] Lustre: Skipped 10 previous similar messages
[ 2800.403794] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2891.293957] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 23:50:26 (1788839426)
[ 2946.724330] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 2949.795581] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3237.010586] LustreError: lustre-OST0000-osc-ffff98afd0025800: operation ldlm_enqueue to node 192.168.203.123@tcp failed: rc = -107
[ 3237.023320] LustreError: Skipped 9 previous similar messages
[ 3282.163689] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3285.035241] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3568.198281] Lustre: lustre-OST0000-osc-ffff98afd0025800: Connection to lustre-OST0000 (at 192.168.203.123@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3568.217329] Lustre: Skipped 3 previous similar messages
[ 3607.189897] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3609.463704] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3863.315546] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 00:06:39 (1788840399)
[ 3879.904335] Lustre: 50968:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788840401/real 1788840401]  req@ffff98afc622ce00 x1875731156744192/t0(0) o101->lustre-MDT0000-mdc-ffff98afd0025800@192.168.203.123@tcp:12/10 lens 576/1152 e 0 to 1 dl 1788840417 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3879.947093] Lustre: 50968:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
[ 3879.988552] Lustre: lustre-MDT0000-mdc-ffff98afd0025800: Connection restored to 192.168.203.123@tcp (at 192.168.203.123@tcp)
[ 3880.004450] Lustre: Skipped 5 previous similar messages
[ 3889.201772] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 00:07:04 (1788840424)
[ 3915.025623] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 00:07:31 (1788840451)
[ 3941.499418] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 00:07:57 (1788840477)
[ 3941.898983] Lustre: Mounted lustre-client - version 2.17.57_113_gf5ca795
[ 3973.599184] Lustre: 2353:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788840494/real 1788840494]  req@ffff98afc622f100 x1875731156763136/t0(0) o400->lustre-MDT0000-mdc-ffff98afc4096000@192.168.203.123@tcp:12/10 lens 224/224 e 0 to 1 dl 1788840510 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3973.600115] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 192.168.203.123@tcp) was lost; in progress operations using this service will fail
[ 3973.629991] Lustre: 2353:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
[ 3973.672610] LustreError: Skipped 2 previous similar messages
[ 3983.855911] Lustre: Evicted from MGS (at 192.168.203.123@tcp) after server handle changed from 0xff98f31736bbfba3 to 0xff98f31736d075d1
[ 3983.865487] Lustre: Skipped 3 previous similar messages
[ 3996.359159] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 3997.845703] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3999.723261] Lustre: Unmounted lustre-client
[ 4005.298901] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 00:09:01 (1788840541)
[ 4126.175608] Lustre: 2353:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788840647/real 1788840647]  req@ffff98afd12a7100 x1875731156796288/t0(0) o4->lustre-OST0000-osc-ffff98afd0025800@192.168.203.123@tcp:6/4 lens 488/448 e 0 to 1 dl 1788840663 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4126.211934] Lustre: 2353:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 59 previous similar messages
[ 4174.175326] Lustre: lustre-OST0000-osc-ffff98afd0025800: Connection to lustre-OST0000 (at 192.168.203.123@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4174.194296] Lustre: Skipped 14 previous similar messages
[ 4302.459674] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 00:13:57 (1788840837)
[ 4353.453353] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 00:14:49 (1788840889)
[ 4357.498841] Lustre: Unmounted lustre-client
[ 4383.493819] Lustre: Mounted lustre-client - version 2.17.57_113_gf5ca795
[ 4390.253730] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 00:15:25 (1788840925)
[ 4390.485687] LustreError: 56895:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4391.535619] LustreError: 56895:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4391.767829] Lustre: *** cfs_fail_loc=305, val=0***
[ 4415.387314] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 00:15:51 (1788840951)
[ 4415.824768] Lustre: Mounted lustre-client - version 2.17.57_113_gf5ca795
[ 4418.662406] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4428.929781] Lustre: Unmounted lustre-client
[ 4436.574929] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 00:16:12 (1788840972)
[ 4521.848738] LustreError: lustre-MDT0000-mdc-ffff98afd0917000: operation mds_reint to node 192.168.203.123@tcp failed: rc = -19
[ 4521.869924] LustreError: Skipped 1 previous similar message
[ 4538.847282] Lustre: 2351:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788841060/real 1788841060]  req@ffff98afc5f0ea00 x1875731159469696/t0(0) o400->MGC192.168.203.123@tcp@192.168.203.123@tcp:26/25 lens 224/224 e 0 to 1 dl 1788841076 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4538.873137] Lustre: 2351:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 78 previous similar messages
[ 4538.888796] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 192.168.203.123@tcp) was lost; in progress operations using this service will fail
[ 4549.096769] Lustre: Evicted from MGS (at 192.168.203.123@tcp) after server handle changed from 0xff98f31736d08f62 to 0xff98f31736d34191
[ 4549.119102] Lustre: MGC192.168.203.123@tcp: Connection restored to 192.168.203.123@tcp (at 192.168.203.123@tcp)
[ 4549.128690] Lustre: Skipped 23 previous similar messages
[ 4550.376207] Lustre: 56291:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.123@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 4710.578858] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 00:20:46 (1788841246)
[ 4716.333996] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4732.393474] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 192.168.203.123@tcp) was lost; in progress operations using this service will fail
[ 4732.440873] Lustre: Evicted from MGS (at 192.168.203.123@tcp) after server handle changed from 0xff98f31736d34191 to 0xff98f31736d80fdf
[ 4732.476825] LustreError: lustre-MDT0000-mdc-ffff98afd0917000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4746.099224] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 00:21:21 (1788841281)
[ 4746.568182] Lustre: Mounted lustre-client - version 2.17.57_113_gf5ca795
[ 4754.122130] LustreError: lustre-OST0000-osc-ffff98afd0917000: operation ldlm_enqueue to node 192.168.203.123@tcp failed: rc = -107
[ 4754.141486] LustreError: lustre-OST0000-osc-ffff98afd0917000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4754.154745] LustreError: 60613:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff98afd0917000: namespace resource [0x240000400:0x3c23:0x0].0x0 (ffff98afc6c7b200) refcount nonzero (2) after lock cleanup; forcing cleanup.
[ 4766.594510] Lustre: Unmounted lustre-client
[ 4775.867704] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 00:21:50 (1788841310)
[ 4780.056646] Lustre: lustre-MDT0000-mdc-ffff98afd0917000: Connection to lustre-MDT0000 (at 192.168.203.123@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4780.069935] Lustre: Skipped 14 previous similar messages
[ 4783.421904] LustreError: lustre-MDT0000-mdc-ffff98afd0917000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4783.442760] LustreError: 61286:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4783.443415] LustreError: 61369:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff98afd0917000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff98afc5f35200) refcount nonzero (2) after lock cleanup; forcing cleanup.
[ 4792.260317] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 00:22:08 (1788841328)
[ 4792.445289] LustreError: 61988:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4796.663504] LustreError: lustre-MDT0000-mdc-ffff98afd0917000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4796.688333] LustreError: 62007:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4796.699562] LustreError: 61988:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=757
[ 4796.721835] LustreError: 61988:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting
[ 4796.764587] LustreError: 62012:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4797.849315] LustreError: 62018:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4797.869095] LustreError: 62018:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4800.170865] LustreError: 62041:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4800.176735] LustreError: 62041:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4804.826368] LustreError: 62086:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4804.842431] LustreError: 62086:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4814.935625] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 00:22:30 (1788841350)
[ 4860.056913] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4861.597981] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4873.712723] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 00:23:29 (1788841409)
[ 4914.508494] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4915.884108] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4926.145958] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 00:24:22 (1788841462)
[ 4992.227648] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4994.027142] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5003.112784] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 00:25:38 (1788841538)
[ 5041.714579] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5043.202515] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5048.872373] Lustre: Unmounted lustre-client
[ 5111.168597] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5114.093307] Lustre: Mounted lustre-client - version 2.17.57_113_gf5ca795
[ 5121.113950] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 00:27:36 (1788841656)
[ 5124.272318] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5126.122038] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 00:27:41 (1788841661)
[ 5160.605351] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 00:28:16 (1788841696)
[ 5162.212324] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5163.600446] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 00:28:19 (1788841699)
[ 5165.013727] Lustre: *** cfs_fail_loc=805, val=0***
[ 5165.061957] Lustre: Mounted lustre-client - version 2.17.57_113_gf5ca795
[ 5170.250194] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5191.647744] Lustre: 2352:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788841712/real 1788841712]  req@ffff98afd1a40000 x1875731161529344/t0(0) o400->lustre-MDT0000-mdc-ffff98afc674a000@192.168.203.123@tcp:12/10 lens 224/224 e 0 to 1 dl 1788841728 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5191.699871] Lustre: 2352:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5191.713379] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 192.168.203.123@tcp) was lost; in progress operations using this service will fail
[ 5191.748800] Lustre: Evicted from MGS (at 192.168.203.123@tcp) after server handle changed from 0xff98f31736d819ef to 0xff98f31736d81cf8
[ 5191.762059] Lustre: MGC192.168.203.123@tcp: Connection restored to 192.168.203.123@tcp (at 192.168.203.123@tcp)
[ 5191.771073] Lustre: Skipped 12 previous similar messages
[ 5201.075590] Lustre: Unmounted lustre-client
[ 5209.002347] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 00:29:04 (1788841744)
[ 5236.596719] LustreError: 2350:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff98afc3ff4e00 x1875731161541120/t94489280516(94489280516) o101->lustre-MDT0000-mdc-ffff98afc674a000@192.168.203.123@tcp:12/10 lens 648/608 e 0 to 0 dl 1788841789 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5246.831817] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5248.593959] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5257.899234] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 00:29:53 (1788841793)
[ 5259.449240] LustreError: lustre-OST0000-osc-ffff98afc674a000: operation ost_write to node 192.168.203.123@tcp failed: rc = -107
[ 5259.455812] LustreError: Skipped 2 previous similar messages
[ 5259.470831] LustreError: lustre-OST0000-osc-ffff98afc674a000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5259.506146] Lustre: 2354:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.123@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5259.568934] LustreError: 73187:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff98afc674a000: namespace resource [0x240000400:0x3c82:0x0].0x0 (ffff98afc6c7b900) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5269.208646] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 00:30:04 (1788841804)
[ 5270.856526] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5272.832216] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 00:30:08 (1788841808)
[ 5274.531363] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5277.225552] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 00:30:12 (1788841812)
[ 5278.893352] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5281.371678] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 00:30:16 (1788841816)
[ 5283.295499] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5285.166566] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 00:30:20 (1788841820)
[ 5286.850750] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5289.375395] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 00:30:24 (1788841824)
[ 5291.060743] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5292.697836] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 00:30:28 (1788841828)
[ 5294.497883] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5296.265824] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 00:30:32 (1788841832)
[ 5297.833024] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5299.498125] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 00:30:35 (1788841835)
[ 5301.285807] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5303.778967] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 00:30:39 (1788841839)
[ 5305.511637] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5307.212469] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 00:30:43 (1788841843)
[ 5308.834039] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5310.588729] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 00:30:46 (1788841846)
[ 5313.172962] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5315.160322] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 00:30:50 (1788841850)
[ 5348.624850] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 00:31:24 (1788841884)
[ 5378.028290] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 00:31:54 (1788841914)
[ 5378.496894] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5388.669707] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 00:32:04 (1788841924)
[ 5388.981182] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5400.012726] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 00:32:15 (1788841935)
[ 5400.382410] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5409.497776] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 00:32:25 (1788841945)
[ 5409.728494] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5417.820781] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 00:32:33 (1788841953)
[ 5418.222572] Lustre: *** cfs_fail_loc=510, val=3***
[ 5426.747651] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 00:32:42 (1788841962)
[ 5427.225139] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5486.495287] Lustre: lustre-OST0001-osc-ffff98afc674a000: Connection to lustre-OST0001 (at 192.168.203.123@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5486.503158] Lustre: Skipped 12 previous similar messages
[ 5495.421776] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 00:33:51 (1788842031)
[ 5495.835705] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5562.070460] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 00:34:58 (1788842098)
[ 5562.463600] LustreError: 83403:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5565.423814] LustreError: lustre-MDT0000-mdc-ffff98afc674a000: operation ldlm_enqueue to node 192.168.203.123@tcp failed: rc = -107
[ 5565.451111] LustreError: lustre-MDT0000-mdc-ffff98afc674a000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5565.459926] LustreError: 83417:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff98afc674a000: inode [0x20000a042:0x8:0x0] mdc close failed: rc = -108
[ 5565.482333] LustreError: 83417:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff98afc674a000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff98afc5cfe600) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5566.535171] LustreError: 83403:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5568.672083] LustreError: 83433:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5571.694553] LustreError: lustre-MDT0000-mdc-ffff98afc674a000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5571.716366] LustreError: 83447:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff98afc674a000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff98afc5fb9c00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5571.749630] LustreError: 83447:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5571.759328] LustreError: 83452:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5571.774988] LustreError: 83452:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5572.767210] LustreError: 83433:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5579.977629] LustreError: lustre-MDT0000-mdc-ffff98afc674a000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5580.479426] LustreError: 83488:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5583.391982] LustreError: lustre-MDT0000-mdc-ffff98afc674a000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5583.499939] LustreError: 83508:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5584.559304] LustreError: 83488:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5584.567506] LustreError: 83488:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff98afc674a000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5585.708671] LustreError: 83525:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5585.716746] LustreError: 83525:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 12 previous similar messages
[ 5590.458042] LustreError: 83563:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5593.430190] LustreError: lustre-MDT0000-mdc-ffff98afc674a000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5593.443575] LustreError: 83577:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff98afc674a000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff98afc5fb9000) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5593.485357] LustreError: 83577:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5594.535522] LustreError: 83563:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5601.443491] LustreError: lustre-MDT0000-mdc-ffff98afc674a000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5601.791699] LustreError: 83619:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5604.688799] LustreError: lustre-MDT0000-mdc-ffff98afc674a000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5605.871184] LustreError: 83619:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5608.934990] LustreError: lustre-MDT0000-mdc-ffff98afc674a000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5608.982357] LustreError: 83666:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5608.993494] LustreError: 83666:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5610.055884] LustreError: 83646:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff98afc674a000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5617.819964] LustreError: 83718:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5617.834358] LustreError: 83718:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5621.443413] LustreError: lustre-MDT0000-mdc-ffff98afc674a000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5621.525536] LustreError: 83740:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5621.536198] LustreError: 83740:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5621.911150] LustreError: 83718:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5621.923542] LustreError: 83718:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5621.939856] LustreError: 83718:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff98afc674a000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5632.589824] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 00:36:08 (1788842168)
[ 5652.341451] LustreError: 80146:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5652.351849] LustreError: 80146:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5656.423616] LustreError: 80146:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5656.434710] LustreError: 80146:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5663.449600] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 00:36:39 (1788842199)
[ 5713.999332] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 00:37:29 (1788842249)
[ 5778.805535] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 00:38:34 (1788842314)
[ 5814.095459] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 00:39:10 (1788842350)
[ 5814.756169] Lustre: Mounted lustre-client - version 2.17.57_113_gf5ca795
[ 5938.563189] Lustre: Unmounted lustre-client
[ 5946.271498] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 00:41:22 (1788842482)
[ 5946.952397] LustreError: 88044:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5949.345508] LustreError: lustre-OST0000-osc-ffff98afc674a000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5949.356536] LustreError: 88131:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff98afc674a000: namespace resource [0x240000400:0x3cad:0x0].0x0 (ffff98afc5f35d00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5949.367448] LustreError: 88131:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 3 previous similar messages
[ 5949.577159] LustreError: 88044:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5949.586629] Lustre: 2352:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.123@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5949.616889] Lustre: lustre-OST0000-osc-ffff98afc674a000: Connection restored to 192.168.203.123@tcp (at 192.168.203.123@tcp)
[ 5949.625296] Lustre: Skipped 18 previous similar messages
[ 5955.641351] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 00:41:31 (1788842491)
[ 6012.896428] Lustre: 88712:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788842495/real 1788842495]  req@ffff98afc62eca80 x1875731161715456/t0(0) o101->lustre-MDT0000-mdc-ffff98afc674a000@192.168.203.123@tcp:12/10 lens 328/344 e 0 to 1 dl 1788842550 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 6012.933389] Lustre: 88712:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 20 previous similar messages
[ 6020.251232] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 00:42:36 (1788842556)
[ 6021.715886] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 6023.346557] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 00:42:39 (1788842559)
[ 6087.677438] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6089.608216] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 00:43:45 (1788842625)
[ 6146.016656] Lustre: lustre-MDT0000-mdc-ffff98afc674a000: Connection to lustre-MDT0000 (at 192.168.203.123@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6146.033416] Lustre: Skipped 16 previous similar messages
[ 6147.844703] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 00:44:43 (1788842683)
[ 6150.441712] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6152.306670] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 00:44:48 (1788842688)
[ 6154.208820] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6156.427350] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 00:44:52 (1788842692)
[ 6193.295829] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 00:45:29 (1788842729)
[ 6204.782133] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6233.080741] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 192.168.203.123@tcp) was lost; in progress operations using this service will fail
[ 6233.127539] LustreError: Skipped 2 previous similar messages
[ 6233.165520] Lustre: Evicted from MGS (at 192.168.203.123@tcp) after server handle changed from 0xff98f31736d8284a to 0xff98f31736d84180
[ 6233.178122] Lustre: Skipped 2 previous similar messages
[ 6246.839784] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6248.666908] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6257.962619] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 00:46:34 (1788842794)
[ 6260.024272] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6261.778234] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 00:46:37 (1788842797)
[ 6279.163796] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 192.168.203.123@tcp) was lost; in progress operations using this service will fail
[ 6279.214353] Lustre: Evicted from MGS (at 192.168.203.123@tcp) after server handle changed from 0xff98f31736d84180 to 0xff98f31736d84538
[ 6291.755737] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 00:47:07 (1788842827)
[ 6321.778614] Lustre: 68654:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.123@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6346.406538] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 00:48:01 (1788842881)
[ 6356.332133] LustreError: lustre-OST0000-osc-ffff98afc674a000: operation ost_setattr to node 192.168.203.123@tcp failed: rc = -107
[ 6356.344841] LustreError: Skipped 9 previous similar messages
[ 6407.759675] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6410.831874] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6496.232834] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 192.168.203.123@tcp) was lost; in progress operations using this service will fail
[ 6496.252892] LustreError: Skipped 1 previous similar message
[ 6496.270861] Lustre: Evicted from MGS (at 192.168.203.123@tcp) after server handle changed from 0xff98f31736d84745 to 0xff98f31736d85599
[ 6496.290111] Lustre: Skipped 1 previous similar message
[ 6497.781070] Lustre: 68654:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.123@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6521.611225] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6523.228450] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6559.128660] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6560.597268] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6682.966475] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 00:53:38 (1788843218)
[ 6740.539675] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6744.091620] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6996.261632] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 00:58:52 (1788843532)
[ 7127.038246] Lustre: lustre-MDT0000-mdc-ffff98afc674a000: Connection to lustre-MDT0000 (at 192.168.203.123@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7127.059150] Lustre: Skipped 7 previous similar messages
[ 7142.382231] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 192.168.203.123@tcp) was lost; in progress operations using this service will fail
[ 7142.394613] LustreError: Skipped 1 previous similar message
[ 7142.421595] Lustre: Evicted from MGS (at 192.168.203.123@tcp) after server handle changed from 0xff98f31736d85a46 to 0xff98f31736e7fcbe
[ 7142.448461] Lustre: Skipped 1 previous similar message
[ 7142.466446] Lustre: MGC192.168.203.123@tcp: Connection restored to 192.168.203.123@tcp (at 192.168.203.123@tcp)
[ 7142.488128] Lustre: Skipped 13 previous similar messages
[ 7175.006865] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 01:01:50 (1788843710)
[ 7176.738311] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7178.327229] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 01:01:54 (1788843714)
[ 7180.705863] LustreError: lustre-MDT0000-mdc-ffff98afc674a000: operation ldlm_enqueue to node 192.168.203.123@tcp failed: rc = -107
[ 7180.719069] LustreError: Skipped 552 previous similar messages
[ 7180.739916] LustreError: lustre-MDT0000-mdc-ffff98afc674a000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7188.133459] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 01:02:04 (1788843724)
[ 7355.370135] LustreError: lustre-OST0000-osc-ffff98afc674a000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7357.496832] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 01:04:53 (1788843893)
[ 7383.519155] Lustre: 2353:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788843900/real 1788843900]  req@ffff98afc8ed3b80 x1875731179485824/t0(0) o4->lustre-OST0000-osc-ffff98afc674a000@192.168.203.123@tcp:6/4 lens 4584/448 e 0 to 1 dl 1788843920 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7383.574722] Lustre: 2353:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 16 previous similar messages
[ 7402.595545] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 01:05:38 (1788843938)
[ 7403.886674] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7405.322883] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 01:05:41 (1788843941)
[ 7406.968252] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7408.437920] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 01:05:44 (1788843944)
[ 7442.738229] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 01:06:18 (1788843978)
[ 7485.922494] Lustre: 2354:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788844007/real 1788844007]  req@ffff98b000d2f480 x1875731179641216/t0(0) o400->MGC192.168.203.123@tcp@192.168.203.123@tcp:26/25 lens 224/224 e 0 to 1 dl 1788844023 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7485.947639] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 192.168.203.123@tcp) was lost; in progress operations using this service will fail
[ 7485.981929] Lustre: Evicted from MGS (at 192.168.203.123@tcp) after server handle changed from 0xff98f31736e7fcbe to 0xff98f31736e8242c
[ 7485.999351] Lustre: Skipped 1 previous similar message
[ 7501.116965] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 01:07:16 (1788844036)
[ 7502.669866] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7504.337171] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 01:07:20 (1788844040)
[ 7506.504599] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7509.064255] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 01:07:24 (1788844044)
[ 7520.827269] Lustre: Unmounted lustre-client
[ 7522.851188] Lustre: Mounted lustre-client - version 2.17.57_113_gf5ca795
[ 7526.198385] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7555.937906] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2