[ 1571.660247] LustreError: lustre-OST0000-osc-ffff8f8511291000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1571.666702] Lustre: 2351:0:(llite_lib.c:4198:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.101@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1574.173507] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 09:54:43 (1781618083)
[ 1575.032417] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1575.622726] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 09:54:45 (1781618085)
[ 1576.248194] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1576.871737] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 09:54:46 (1781618086)
[ 1595.361402] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 192.168.202.101@tcp) was lost; in progress operations using this service will fail
[ 1595.381777] Lustre: Evicted from MGS (at 192.168.202.101@tcp) after server handle changed from 0x9ffb0719ff7983d5 to 0x9ffb0719ff79a59a
[ 1687.702243] LustreError: lustre-MDT0000-mdc-ffff8f8511291000: operation mds_reint to node 192.168.202.101@tcp failed: rc = -19
[ 1687.722189] LustreError: Skipped 2 previous similar messages
[ 1706.982931] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 192.168.202.101@tcp) was lost; in progress operations using this service will fail
[ 1707.010446] Lustre: Evicted from MGS (at 192.168.202.101@tcp) after server handle changed from 0x9ffb0719ff79a59a to 0x9ffb0719ff7d1459
[ 1708.846686] Lustre: 15865:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.202.101@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 1731.435763] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 09:57:20 (1781618240)
[ 1731.674524] Lustre: *** cfs_fail_loc=305, val=0***
[ 1731.676359] Lustre: Skipped 4 previous similar messages
[ 1768.421815] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 192.168.202.101@tcp) was lost; in progress operations using this service will fail
[ 1768.453543] Lustre: Evicted from MGS (at 192.168.202.101@tcp) after server handle changed from 0x9ffb0719ff7d1459 to 0x9ffb0719ff7d1c01
[ 1775.113928] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1776.130408] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1781.966723] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 09:58:11 (1781618291)
[ 1788.902587] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 192.168.202.101@tcp) was lost; in progress operations using this service will fail
[ 1788.914018] Lustre: Evicted from MGS (at 192.168.202.101@tcp) after server handle changed from 0x9ffb0719ff7d1c01 to 0x9ffb0719ff7d1f26
[ 1790.323783] LustreError: lustre-MDT0000-mdc-ffff8f8511291000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 1803.707606] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 09:58:33 (1781618313)
[ 1811.643518] LustreError: lustre-OST0000-osc-ffff8f8511291000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1823.044433] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 09:58:52 (1781618332)
[ 1835.324388] LustreError: lustre-MDT0000-mdc-ffff8f8511291000: operation ldlm_enqueue to node 192.168.202.101@tcp failed: rc = -19
[ 1855.466116] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 192.168.202.101@tcp) was lost; in progress operations using this service will fail
[ 1855.505415] Lustre: Evicted from MGS (at 192.168.202.101@tcp) after server handle changed from 0x9ffb0719ff7d1f26 to 0x9ffb0719ff7d90f4
[ 1860.322618] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1861.670904] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1925.255441] Lustre: lustre-MDT0000-mdc-ffff8f8511291000: Connection to lustre-MDT0000 (at 192.168.202.101@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 1925.267630] Lustre: Skipped 16 previous similar messages
[ 1942.511979] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 192.168.202.101@tcp) was lost; in progress operations using this service will fail
[ 1942.538813] Lustre: Evicted from MGS (at 192.168.202.101@tcp) after server handle changed from 0x9ffb0719ff7d90f4 to 0x9ffb0719ff802b28
[ 1942.556984] Lustre: MGC192.168.202.101@tcp: Connection restored to 192.168.202.101@tcp (at 192.168.202.101@tcp)
[ 1942.560663] Lustre: Skipped 21 previous similar messages
[ 1952.274913] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1953.627419] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2021.533112] LustreError: lustre-MDT0000-mdc-ffff8f8511291000: operation mds_close to node 192.168.202.101@tcp failed: rc = -19
[ 2021.543331] LustreError: Skipped 1 previous similar message
[ 2040.607464] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 192.168.202.101@tcp) was lost; in progress operations using this service will fail
[ 2049.790062] Lustre: Evicted from MGS (at 192.168.202.101@tcp) after server handle changed from 0x9ffb0719ff802b28 to 0x9ffb0719ff81a376
[ 2067.101258] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2070.300779] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2099.698626] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 10:03:28 (1781618608)
[ 2123.756170] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 192.168.202.101@tcp) was lost; in progress operations using this service will fail
[ 2123.779645] Lustre: Evicted from MGS (at 192.168.202.101@tcp) after server handle changed from 0x9ffb0719ff81a376 to 0x9ffb0719ff8282f8
[ 2133.489055] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2155.487645] Lustre: 2349:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781618649/real 1781618649]  req@ffff8f8511280a80 x1868160413058048/t0(0) o400->MGC192.168.202.101@tcp@192.168.202.101@tcp:26/25 lens 224/224 e 0 to 1 dl 1781618665 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2155.506138] Lustre: 2349:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 9 previous similar messages
[ 2164.691802] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2198.097538] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2242.317878] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2287.391274] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 192.168.202.101@tcp) was lost; in progress operations using this service will fail
[ 2287.419419] LustreError: Skipped 3 previous similar messages
[ 2287.458074] Lustre: Evicted from MGS (at 192.168.202.101@tcp) after server handle changed from 0x9ffb0719ff830fe6 to 0x9ffb0719ff83d456
[ 2287.480454] Lustre: Skipped 3 previous similar messages
[ 2297.753954] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2326.355868] LustreError: lustre-MDT0000-mdc-ffff8f8511291000: operation ldlm_enqueue to node 192.168.202.101@tcp failed: rc = -19
[ 2326.365648] LustreError: Skipped 8 previous similar messages
[ 2347.517325] Lustre: 15865:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.202.101@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2363.150805] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2454.784521] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 10:09:23 (1781618963)
[ 2507.750876] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2510.581076] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 2801.259224] Lustre: lustre-OST0000-osc-ffff8f8511291000: Connection to lustre-OST0000 (at 192.168.202.101@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2801.271570] Lustre: Skipped 8 previous similar messages
[ 2846.816434] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2849.682143] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3131.153845] LustreError: lustre-OST0000-osc-ffff8f8511291000: operation ldlm_enqueue to node 192.168.202.101@tcp failed: rc = -107
[ 3131.160343] LustreError: Skipped 4 previous similar messages
[ 3150.354594] Lustre: lustre-OST0000-osc-ffff8f8511291000: Connection restored to 192.168.202.101@tcp (at 192.168.202.101@tcp)
[ 3150.365949] Lustre: Skipped 17 previous similar messages
[ 3163.741175] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3165.598109] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3424.927645] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 10:25:34 (1781619934)
[ 3442.655165] Lustre: 50421:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781619936/real 1781619936]  req@ffff8f8508619180 x1868160433565696/t0(0) o101->lustre-MDT0000-mdc-ffff8f8511291000@192.168.202.101@tcp:12/10 lens 576/1152 e 0 to 1 dl 1781619952 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3442.694816] Lustre: 50421:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 3442.704948] Lustre: lustre-MDT0000-mdc-ffff8f8511291000: Connection to lustre-MDT0000 (at 192.168.202.101@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3442.722131] Lustre: Skipped 1 previous similar message
[ 3449.198608] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 10:25:58 (1781619958)
[ 3474.141867] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 10:26:23 (1781619983)
[ 3497.546555] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 10:26:46 (1781620006)
[ 3497.935765] Lustre: Mounted lustre-client
[ 3528.677695] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 192.168.202.101@tcp) was lost; in progress operations using this service will fail
[ 3528.686903] LustreError: Skipped 2 previous similar messages
[ 3528.707673] Lustre: Evicted from MGS (at 192.168.202.101@tcp) after server handle changed from 0x9ffb0719ff84ff34 to 0x9ffb0719ff9abde6
[ 3528.718879] Lustre: Skipped 2 previous similar messages
[ 3529.696095] Lustre: 2352:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781620023/real 1781620023]  req@ffff8f8508f00000 x1868160433584512/t0(0) o400->lustre-MDT0000-mdc-ffff8f85100c6000@192.168.202.101@tcp:12/10 lens 224/224 e 0 to 1 dl 1781620039 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3529.725380] Lustre: 2352:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 3538.468356] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3540.116922] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3542.218882] Lustre: Unmounted lustre-client
[ 3547.545576] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 10:27:36 (1781620056)
[ 3681.759156] Lustre: 2349:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781620176/real 1781620176]  req@ffff8f85087ee680 x1868160433616384/t0(0) o4->lustre-OST0000-osc-ffff8f8511291000@192.168.202.101@tcp:6/4 lens 488/448 e 0 to 1 dl 1781620192 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 3681.801909] Lustre: 2349:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 56 previous similar messages
[ 3762.666645] Lustre: lustre-OST0000-osc-ffff8f8511291000: Connection restored to 192.168.202.101@tcp (at 192.168.202.101@tcp)
[ 3762.673740] Lustre: Skipped 17 previous similar messages
[ 3834.807832] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 10:32:24 (1781620344)
[ 3880.251851] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 10:33:09 (1781620389)
[ 3882.196190] Lustre: Unmounted lustre-client
[ 3896.484523] Lustre: Mounted lustre-client
[ 3900.391732] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 10:33:29 (1781620409)
[ 3900.738709] LustreError: 56321:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 3901.783107] LustreError: 56321:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 3901.933403] Lustre: *** cfs_fail_loc=305, val=0***
[ 3922.350662] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 10:33:51 (1781620431)
[ 3922.640848] Lustre: Mounted lustre-client
[ 3923.693085] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 3933.962610] Lustre: Unmounted lustre-client
[ 3938.082759] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 10:34:07 (1781620447)
[ 3995.676333] LustreError: lustre-MDT0000-mdc-ffff8f8505d2f800: operation mds_reint to node 192.168.202.101@tcp failed: rc = -19
[ 4015.076296] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 192.168.202.101@tcp) was lost; in progress operations using this service will fail
[ 4015.082675] Lustre: Evicted from MGS (at 192.168.202.101@tcp) after server handle changed from 0x9ffb0719ff9ad563 to 0x9ffb0719ff9d9488
[ 4099.515827] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 10:36:49 (1781620609)
[ 4102.015469] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4107.233752] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 192.168.202.101@tcp) was lost; in progress operations using this service will fail
[ 4107.239716] Lustre: lustre-MDT0000-mdc-ffff8f8505d2f800: Connection to lustre-MDT0000 (at 192.168.202.101@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4107.241667] Lustre: Evicted from MGS (at 192.168.202.101@tcp) after server handle changed from 0x9ffb0719ff9d9488 to 0x9ffb0719ffa24d04
[ 4107.244574] Lustre: Skipped 24 previous similar messages
[ 4107.262000] LustreError: lustre-MDT0000-mdc-ffff8f8505d2f800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4115.847024] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 10:37:05 (1781620625)
[ 4116.102392] Lustre: Mounted lustre-client
[ 4121.577339] LustreError: lustre-OST0000-osc-ffff8f8505d2f800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4121.582673] LustreError: 59996:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-OST0000-osc-ffff8f8505d2f800: namespace resource [0x240000400:0x4443:0x0].0x0 (ffff8f8507ceb000) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4132.831260] Lustre: 2351:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781620627/real 1781620627]  req@ffff8f850874a300 x1868160438253824/t0(0) o103->lustre-OST0000-osc-ffff8f8504e25800@192.168.202.101@tcp:17/18 lens 328/224 e 0 to 1 dl 1781620643 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295
[ 4132.849219] Lustre: 2351:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 81 previous similar messages
[ 4133.371502] Lustre: Unmounted lustre-client
[ 4136.683336] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 10:37:26 (1781620646)
[ 4142.062944] LustreError: lustre-MDT0000-mdc-ffff8f8505d2f800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4142.069851] LustreError: 60668:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4145.433119] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 10:37:35 (1781620655)
[ 4145.562523] LustreError: 61370:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4149.630763] LustreError: lustre-MDT0000-mdc-ffff8f8505d2f800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4149.634703] LustreError: 61387:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4149.638769] LustreError: 61370:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=927
[ 4149.648810] LustreError: 61370:0:(import.c:719:ptlrpc_connect_import_locked()) already connecting
[ 4149.663377] LustreError: 61392:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4150.731196] LustreError: 61398:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4150.736871] LustreError: 61398:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4152.904084] LustreError: 61422:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4152.909811] LustreError: 61422:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4157.186410] LustreError: 61466:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4157.189548] LustreError: 61466:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4163.416739] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 10:37:53 (1781620673)
[ 4187.093354] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4187.990663] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4192.829699] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 10:38:22 (1781620702)
[ 4214.045308] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4214.825298] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4219.545451] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 10:38:49 (1781620729)
[ 4265.457527] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4266.053161] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4269.841935] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 10:39:39 (1781620779)
[ 4290.521478] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4291.134498] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4293.656558] Lustre: Unmounted lustre-client
[ 4320.290212] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4321.871939] Lustre: Mounted lustre-client
[ 4324.763923] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 10:40:34 (1781620834)
[ 4325.829608] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 4326.571879] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 10:40:36 (1781620836)
[ 4339.734256] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 10:40:49 (1781620849)
[ 4340.366744] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 4341.019871] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 10:40:50 (1781620850)
[ 4341.584865] Lustre: *** cfs_fail_loc=805, val=0***
[ 4341.600153] Lustre: Mounted lustre-client
[ 4343.856483] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4361.089069] LustreError: 2348:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8f85044e5800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff8f8507978300) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4361.633262] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 192.168.202.101@tcp) was lost; in progress operations using this service will fail
[ 4361.640626] Lustre: Evicted from MGS (at 192.168.202.101@tcp) after server handle changed from 0x9ffb0719ffa2577d to 0x9ffb0719ffa25d6c
[ 4361.659966] Lustre: Unmounted lustre-client
[ 4364.888498] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 10:41:14 (1781620874)
[ 4381.681048] Lustre: MGC192.168.202.101@tcp: Connection restored to 192.168.202.101@tcp (at 192.168.202.101@tcp)
[ 4381.685070] Lustre: Skipped 23 previous similar messages
[ 4381.716436] LustreError: 2348:0:(client.c:3447:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff8f8509970a80 x1868160438321024/t90194313219(90194313219) o101->lustre-MDT0000-mdc-ffff8f8505d0c000@192.168.202.101@tcp:12/10 lens 648/608 e 0 to 0 dl 1781620907 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 4385.991853] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4386.657752] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4390.747674] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 10:41:40 (1781620900)
[ 4391.451904] LustreError: lustre-OST0000-osc-ffff8f8505d0c000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4391.457392] Lustre: 2350:0:(llite_lib.c:4198:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.101@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 4391.470330] LustreError: 72467:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-OST0000-osc-ffff8f8505d0c000: namespace resource [0x240000400:0x44a2:0x0].0x0 (ffff8f85041e7300) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4391.480079] LustreError: 72467:0:(ldlm_resource.c:1180:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4395.317735] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 10:41:45 (1781620905)
[ 4396.040282] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 4396.683873] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 10:41:46 (1781620906)
[ 4397.306066] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 4397.959508] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 10:41:47 (1781620907)
[ 4398.574330] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 4399.244744] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 10:41:49 (1781620909)
[ 4399.826292] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 4400.491677] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 10:41:50 (1781620910)
[ 4401.145495] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 4401.841421] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 10:41:51 (1781620911)
[ 4402.442732] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 4403.089828] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 10:41:52 (1781620912)
[ 4403.685539] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 4404.349387] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 10:41:54 (1781620914)
[ 4405.066530] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 4405.764925] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 10:41:55 (1781620915)
[ 4406.431988] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 4407.178383] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 10:41:56 (1781620916)
[ 4407.884662] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 4408.609060] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 10:41:58 (1781620918)
[ 4409.246269] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 4410.004348] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 10:41:59 (1781620919)
[ 4411.055695] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 4411.826839] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 10:42:01 (1781620921)
[ 4423.854222] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 10:42:13 (1781620933)
[ 4452.786450] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 10:42:42 (1781620962)
[ 4453.835264] Lustre: *** cfs_fail_loc=51b, val=3***
[ 4463.702680] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 10:42:52 (1781620972)
[ 4464.479955] Lustre: *** cfs_fail_loc=51b, val=4***
[ 4475.010911] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 10:43:03 (1781620983)
[ 4475.257221] Lustre: *** cfs_fail_loc=50f, val=3***
[ 4485.384970] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 10:43:14 (1781620994)
[ 4485.719938] Lustre: *** cfs_fail_loc=50f, val=4***
[ 4494.607723] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 10:43:23 (1781621003)
[ 4495.122946] Lustre: *** cfs_fail_loc=510, val=3***
[ 4504.793672] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 10:43:33 (1781621013)
[ 4505.183661] Lustre: *** cfs_fail_loc=51b, val=3***
[ 4569.093712] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 10:44:38 (1781621078)
[ 4569.546341] Lustre: *** cfs_fail_loc=51c, val=3***
[ 4632.745810] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 10:45:42 (1781621142)
[ 4633.041303] LustreError: 82659:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 4635.598462] LustreError: lustre-MDT0000-mdc-ffff8f8505d0c000: operation ldlm_enqueue to node 192.168.202.101@tcp failed: rc = -107
[ 4635.608360] LustreError: Skipped 5 previous similar messages
[ 4635.617477] LustreError: lustre-MDT0000-mdc-ffff8f8505d0c000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4635.636568] LustreError: 82673:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8f8505d0c000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8f8503777b00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4637.111459] LustreError: 82659:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 4639.213288] LustreError: 82689:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 4641.846526] LustreError: lustre-MDT0000-mdc-ffff8f8505d0c000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4641.858280] LustreError: 82703:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8f8505d0c000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8f85041e7a00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4641.890583] LustreError: 82703:0:(ldlm_resource.c:1180:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4643.280492] LustreError: 82689:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 4649.975293] LustreError: lustre-MDT0000-mdc-ffff8f8505d0c000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4650.278197] LustreError: 82744:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 4652.980995] LustreError: lustre-MDT0000-mdc-ffff8f8505d0c000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4653.044740] LustreError: 82764:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4653.053397] LustreError: 82764:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 20 previous similar messages
[ 4654.351559] LustreError: 82744:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 4654.358757] LustreError: 82744:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8f8505d0c000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 4659.884499] LustreError: 82819:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 4662.547242] LustreError: lustre-MDT0000-mdc-ffff8f8505d0c000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4662.564541] LustreError: 82834:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8f8505d0c000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8f8503777e00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4662.588134] LustreError: 82834:0:(ldlm_resource.c:1180:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4663.943252] LustreError: 82819:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 4670.488906] LustreError: lustre-MDT0000-mdc-ffff8f8505d0c000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4670.780981] LustreError: 82875:0:(ldlm_flock.c:858:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 4673.396799] LustreError: lustre-MDT0000-mdc-ffff8f8505d0c000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4674.855121] LustreError: 82875:0:(ldlm_flock.c:858:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 4677.478143] LustreError: lustre-MDT0000-mdc-ffff8f8505d0c000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4677.515554] LustreError: 82923:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4677.521231] LustreError: 82923:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 4678.991675] LustreError: 82903:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8f8505d0c000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 4686.286335] LustreError: 82975:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 4686.296203] LustreError: 82975:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 4689.319441] LustreError: lustre-MDT0000-mdc-ffff8f8505d0c000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4690.367181] LustreError: 82975:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 4690.371864] LustreError: 82975:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 4690.379744] LustreError: 82975:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8f8505d0c000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 4697.500264] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 10:46:47 (1781621207)
[ 4713.951227] Lustre: lustre-MDT0000-mdc-ffff8f8505d0c000: Connection to lustre-MDT0000 (at 192.168.202.101@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4713.958296] Lustre: Skipped 27 previous similar messages
[ 4715.241380] LustreError: 67966:0:(ldlm_lockd.c:2883:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 4715.245653] LustreError: 67966:0:(ldlm_lockd.c:2883:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 4719.303432] LustreError: 67966:0:(ldlm_lockd.c:2883:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 4719.308048] LustreError: 67966:0:(ldlm_lockd.c:2883:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 4724.042400] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 10:47:13 (1781621233)
[ 4766.688231] Lustre: 84423:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781621235/real 1781621235]  req@ffff8f8509973800 x1868160438439296/t0(0) o101->lustre-MDT0000-mdc-ffff8f8505d0c000@192.168.202.101@tcp:12/10 lens 576/1152 e 1 to 1 dl 1781621276 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'stat.0' uid:0 gid:0 projid:0
[ 4766.717069] Lustre: 84423:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 12 previous similar messages
[ 4771.736907] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 10:48:01 (1781621281)
[ 4831.975366] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 10:49:01 (1781621341)
[ 4860.681205] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 10:49:30 (1781621370)
[ 4861.015351] Lustre: Mounted lustre-client
[ 4982.789092] Lustre: Unmounted lustre-client
[ 4986.299784] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 10:51:35 (1781621495)
[ 4986.743089] LustreError: 87298:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 4988.175279] LustreError: lustre-OST0000-osc-ffff8f8505d0c000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4988.182392] LustreError: 87386:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-OST0000-osc-ffff8f8505d0c000: namespace resource [0x240000400:0x44cd:0x0].0x0 (ffff8f85040f7500) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4988.189471] LustreError: 87386:0:(ldlm_resource.c:1180:ldlm_resource_complain()) Skipped 3 previous similar messages
[ 4988.312531] LustreError: 87298:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout interrupted
[ 4988.315153] Lustre: 2350:0:(llite_lib.c:4198:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.101@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 4988.323075] Lustre: lustre-OST0000-osc-ffff8f8505d0c000: Connection restored to 192.168.202.101@tcp (at 192.168.202.101@tcp)
[ 4988.327848] Lustre: Skipped 15 previous similar messages
[ 4991.284316] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 10:51:40 (1781621500)
[ 5050.780183] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 10:52:40 (1781621560)
[ 5051.339833] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 5051.972235] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 10:52:41 (1781621561)
[ 5110.307103] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 5110.978645] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 10:53:40 (1781621620)
[ 5166.618821] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 10:54:36 (1781621676)
[ 5167.738249] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 5168.483071] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 10:54:38 (1781621678)
[ 5169.123392] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 5169.946420] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 10:54:39 (1781621679)
[ 5183.088246] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 10:54:52 (1781621692)
[ 5187.779175] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5207.008318] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 192.168.202.101@tcp) was lost; in progress operations using this service will fail
[ 5207.016401] LustreError: Skipped 2 previous similar messages
[ 5207.029198] Lustre: Evicted from MGS (at 192.168.202.101@tcp) after server handle changed from 0x9ffb0719ffa265a0 to 0x9ffb0719ffa27fbd
[ 5207.034059] Lustre: Skipped 2 previous similar messages
[ 5210.101756] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5210.795383] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5215.551386] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 10:55:25 (1781621725)
[ 5216.451216] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 5217.095094] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 10:55:26 (1781621726)
[ 5227.271233] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 10:55:36 (1781621736)
[ 5244.798482] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 10:55:54 (1781621754)
[ 5247.299290] LustreError: lustre-OST0000-osc-ffff8f8505d0c000: operation ost_setattr to node 192.168.202.101@tcp failed: rc = -107
[ 5247.306802] LustreError: Skipped 9 previous similar messages
[ 5270.575372] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5271.688326] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5335.009619] Lustre: lustre-MDT0000-mdc-ffff8f8505d0c000: Connection to lustre-MDT0000 (at 192.168.202.101@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5335.013853] Lustre: Skipped 10 previous similar messages
[ 5363.118949] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5364.888824] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5397.476328] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5399.816413] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5426.720220] Lustre: 2350:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781621881/real 1781621881]  req@ffff8f850450b480 x1868160444995968/t0(0) o400->lustre-MDT0000-mdc-ffff8f8505d0c000@192.168.202.101@tcp:12/10 lens 224/224 e 0 to 1 dl 1781621936 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5426.750967] Lustre: 2350:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 9 previous similar messages
[ 5517.932404] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 11:00:27 (1781622027)
[ 5565.783171] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5568.629125] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5834.615906] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 11:05:43 (1781622343)
[ 6026.737711] Lustre: lustre-MDT0000-mdc-ffff8f8505d0c000: Connection to lustre-MDT0000 (at 192.168.202.101@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6026.756159] Lustre: Skipped 2 previous similar messages
[ 6026.767153] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 192.168.202.101@tcp) was lost; in progress operations using this service will fail
[ 6026.778175] LustreError: Skipped 4 previous similar messages
[ 6026.805428] Lustre: Evicted from MGS (at 192.168.202.101@tcp) after server handle changed from 0x9ffb0719ffa298d7 to 0x9ffb0719ffb195c5
[ 6026.814707] Lustre: Skipped 4 previous similar messages
[ 6026.842576] Lustre: MGC192.168.202.101@tcp: Connection restored to 192.168.202.101@tcp (at 192.168.202.101@tcp)
[ 6026.858368] Lustre: Skipped 13 previous similar messages
[ 6049.119208] Lustre: 2351:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781622526/real 1781622526]  req@ffff8f8509443b80 x1868160455379968/t0(0) o400->lustre-MDT0000-mdc-ffff8f8505d0c000@192.168.202.101@tcp:12/10 lens 224/224 e 0 to 1 dl 1781622559 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6049.168694] Lustre: 2351:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 6060.254929] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 11:09:28 (1781622568)
[ 6062.521431] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 6064.610655] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 11:09:33 (1781622573)
[ 6066.657756] LustreError: lustre-MDT0000-mdc-ffff8f8505d0c000: operation ldlm_enqueue to node 192.168.202.101@tcp failed: rc = -107
[ 6066.670382] LustreError: Skipped 10 previous similar messages
[ 6066.686397] LustreError: lustre-MDT0000-mdc-ffff8f8505d0c000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6075.496289] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 11:09:44 (1781622584)
[ 6243.308699] LustreError: lustre-OST0000-osc-ffff8f8505d0c000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6245.295615] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 11:12:34 (1781622754)
[ 6289.007795] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 11:13:17 (1781622797)
[ 6290.320694] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 6292.389640] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 11:13:21 (1781622801)
[ 6294.070652] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 6296.183909] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 11:13:24 (1781622804)
[ 6333.130787] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 11:14:02 (1781622842)
[ 6387.601517] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 11:14:56 (1781622896)
[ 6389.200718] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 6391.115217] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 11:15:00 (1781622900)
[ 6392.646098] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 6394.706497] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 11:15:03 (1781622903)
[ 6410.918915] Lustre: Unmounted lustre-client
[ 6413.475668] Lustre: Mounted lustre-client
[ 6417.688238] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6440.823475] Lustre: 106326:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.202.101@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6456.723964] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2
[ 6471.596294] Lustre: DEBUG MARKER: == recovery-small test 156: tot_granted miscount after client eviction ========================================================== 11:16:20 (1781622980)
[ 6478.703694] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-OST0000
[ 6499.834959] LustreError: 2348:0:(client.c:3532:ptlrpc_replay_req()) cfs_fail_timeout id 536 sleeping for 45000ms
[ 6544.887386] LustreError: 2348:0:(client.c:3532:ptlrpc_replay_req()) cfs_fail_timeout id 536 awake
[ 6544.919186] LustreError: lustre-OST0000-osc-ffff8f85100c1000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6551.141825] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 6553.005768] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6563.711874] Lustre: DEBUG MARKER: == recovery-small test 157: eviction during mmaped i/o === 11:17:52 (1781623072)
[ 6564.258619] LustreError: 108846:0:(vvp_io.c:1473:vvp_io_fault_start()) cfs_fail_timeout id 1432 sleeping for 3000ms
[ 6567.279163] LustreError: 108846:0:(vvp_io.c:1473:vvp_io_fault_start()) cfs_fail_timeout id 1432 awake
[ 6567.332373] LustreError: lustre-OST0000-osc-ffff8f85100c1000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6567.355422] LustreError: 108860:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-OST0000-osc-ffff8f85100c1000: namespace resource [0x240000401:0x22c3:0x0].0x0 (ffff8f85050dcc00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6574.005820] Lustre: DEBUG MARKER: == recovery-small test 158a: connect without access right ========================================================== 11:18:02 (1781623082)
[ 6575.680897] Lustre: DEBUG MARKER: SKIP: recovery-small test_158a needs >= 2 MDTS
[ 6577.197682] Lustre: DEBUG MARKER: == recovery-small test 160: MDT destroys are blocked by grouplocks ========================================================== 11:18:06 (1781623086)
[ 6580.602344] LustreError: lustre-MDT0000-mdc-ffff8f85100c1000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6628.127058] Lustre: DEBUG MARKER: == recovery-small test 161: evict osp by ping evictor ==== 11:18:56 (1781623136)
[ 6629.404931] Lustre: DEBUG MARKER: SKIP: recovery-small test_161 needs >= 2 MDTs
[ 6631.002401] Lustre: DEBUG MARKER: == recovery-small test 162: File attributes should be persisted after MDS failover ========================================================== 11:18:59 (1781623139)
[ 6642.155044] Lustre: lustre-MDT0000-mdc-ffff8f85100c1000: Connection to lustre-MDT0000 (at 192.168.202.101@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6642.173059] Lustre: Skipped 8 previous similar messages
[ 6642.179041] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 192.168.202.101@tcp) was lost; in progress operations using this service will fail
[ 6642.207705] LustreError: Skipped 2 previous similar messages
[ 6642.242511] Lustre: Evicted from MGS (at 192.168.202.101@tcp) after server handle changed from 0x9ffb0719ffb1cb8e to 0x9ffb0719ffb1e0e9
[ 6642.251096] Lustre: Skipped 3 previous similar messages
[ 6642.257945] LustreError: 2348:0:(client.c:3447:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff8f850962ad80 x1868160455683968/t133143986232(133143986232) o101->lustre-MDT0000-mdc-ffff8f85100c1000@192.168.202.101@tcp:12/10 lens 576/608 e 0 to 0 dl 1781623187 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'chattr.0' uid:0 gid:0 projid:0
[ 6642.261602] Lustre: MGC192.168.202.101@tcp: Connection restored to 192.168.202.101@tcp (at 192.168.202.101@tcp)
[ 6642.286826] Lustre: Skipped 9 previous similar messages
[ 6656.689882] Lustre: DEBUG MARKER: == recovery-small test 163: changelog check for fail write and processing records ========================================================== 11:19:25 (1781623165)
[ 6671.839845] Lustre: 2349:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781623147/real 1781623147]  req@ffff8f8506d38a80 x1868160455689600/t0(0) o400->lustre-MDT0000-mdc-ffff8f85100c1000@192.168.202.101@tcp:12/10 lens 224/224 e 0 to 1 dl 1781623182 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6671.886393] Lustre: 2349:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 9 previous similar messages
[ 6675.293156] Lustre: DEBUG MARKER: == recovery-small test 170: Reconnect after REPLAY_LOCKS hangs (LU-18154) ========================================================== 11:19:44 (1781623184)
[ 6679.660727] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-OST0000
[ 6714.449955] Lustre: *** cfs_fail_loc=537, val=0***
[ 6714.451635] LustreError: 2348:0:(import.c:719:ptlrpc_connect_import_locked()) already connecting
[ 6714.710569] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount REPLAY_LOCKS osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 8282.565451] Lustre: DEBUG MARKER: rpc test_170: @@@@@@ FAIL: can't put import for osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid into REPLAY_LOCKS state after 1475 sec, have IDLE