[ 1928.494191] LustreError: lustre-OST0000-osc-ffff8f0709a47800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1928.519038] Lustre: 2360:0:(llite_lib.c:4339:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.121@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1933.544587] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 02:50:36 (1785221436)
[ 1935.019454] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1936.287966] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 02:50:39 (1785221439)
[ 1937.401872] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1938.586551] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 02:50:41 (1785221441)
[ 1959.407062] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 192.168.204.121@tcp) was lost; in progress operations using this service will fail
[ 1959.423918] Lustre: Evicted from MGS (at 192.168.204.121@tcp) after server handle changed from 0x7b8a9083d9b5b654 to 0x7b8a9083d9b5d1a5
[ 2056.863491] LustreError: lustre-MDT0000-mdc-ffff8f0709a47800: operation ldlm_enqueue to node 192.168.204.121@tcp failed: rc = -19
[ 2056.876250] LustreError: Skipped 3 previous similar messages
[ 2072.037886] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 192.168.204.121@tcp) was lost; in progress operations using this service will fail
[ 2072.057397] Lustre: Evicted from MGS (at 192.168.204.121@tcp) after server handle changed from 0x7b8a9083d9b5d1a5 to 0x7b8a9083d9b8bc98
[ 2081.493784] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 02:53:04 (1785221584)
[ 2081.753590] Lustre: *** cfs_fail_loc=305, val=0***
[ 2081.755253] Lustre: Skipped 2 previous similar messages
[ 2118.118165] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 192.168.204.121@tcp) was lost; in progress operations using this service will fail
[ 2118.137718] Lustre: Evicted from MGS (at 192.168.204.121@tcp) after server handle changed from 0x7b8a9083d9b8bc98 to 0x7b8a9083d9b8c439
[ 2127.576475] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2128.798294] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2134.638655] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 02:53:58 (1785221638)
[ 2143.728105] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 192.168.204.121@tcp) was lost; in progress operations using this service will fail
[ 2143.744215] Lustre: Evicted from MGS (at 192.168.204.121@tcp) after server handle changed from 0x7b8a9083d9b8c439 to 0x7b8a9083d9b8c8bc
[ 2143.774261] LustreError: lustre-MDT0000-mdc-ffff8f0709a47800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2157.515766] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 02:54:20 (1785221660)
[ 2165.426653] LustreError: lustre-OST0000-osc-ffff8f0709a47800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2177.517548] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 02:54:40 (1785221680)
[ 2189.235558] LustreError: lustre-MDT0000-mdc-ffff8f0709a47800: operation ldlm_enqueue to node 192.168.204.121@tcp failed: rc = -19
[ 2189.242870] LustreError: Skipped 1 previous similar message
[ 2205.158435] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 192.168.204.121@tcp) was lost; in progress operations using this service will fail
[ 2205.173834] Lustre: Evicted from MGS (at 192.168.204.121@tcp) after server handle changed from 0x7b8a9083d9b8c8bc to 0x7b8a9083d9b931a0
[ 2214.193815] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2215.350050] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2278.991566] Lustre: lustre-MDT0000-mdc-ffff8f0709a47800: Connection to lustre-MDT0000 (at 192.168.204.121@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2279.008147] Lustre: Skipped 15 previous similar messages
[ 2297.323793] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 192.168.204.121@tcp) was lost; in progress operations using this service will fail
[ 2297.348669] Lustre: Evicted from MGS (at 192.168.204.121@tcp) after server handle changed from 0x7b8a9083d9b931a0 to 0x7b8a9083d9bbaff7
[ 2297.369502] Lustre: MGC192.168.204.121@tcp: Connection restored to 192.168.204.121@tcp (at 192.168.204.121@tcp)
[ 2297.378392] Lustre: Skipped 20 previous similar messages
[ 2305.414749] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2306.528561] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2370.478487] LustreError: lustre-MDT0000-mdc-ffff8f0709a47800: operation mds_reint to node 192.168.204.121@tcp failed: rc = -19
[ 2370.482463] LustreError: Skipped 1 previous similar message
[ 2389.483618] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 192.168.204.121@tcp) was lost; in progress operations using this service will fail
[ 2389.500801] Lustre: Evicted from MGS (at 192.168.204.121@tcp) after server handle changed from 0x7b8a9083d9bbaff7 to 0x7b8a9083d9be2ce9
[ 2396.198526] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2397.546140] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2428.938460] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 02:58:51 (1785221931)
[ 2450.911565] Lustre: 2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785221939/real 1785221939]  req@ffff8f0719ec2300 x1871938450460288/t0(0) o400->MGC192.168.204.121@tcp@192.168.204.121@tcp:26/25 lens 224/224 e 0 to 1 dl 1785221955 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2450.978166] Lustre: 2358:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
[ 2461.159687] Lustre: Evicted from MGS (at 192.168.204.121@tcp) after server handle changed from 0x7b8a9083d9be2ce9 to 0x7b8a9083d9bf314e
[ 2462.449763] Lustre: 15907:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.204.121@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2480.133629] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2507.231460] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 192.168.204.121@tcp) was lost; in progress operations using this service will fail
[ 2507.259066] LustreError: Skipped 1 previous similar message
[ 2525.209452] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2553.453839] Lustre: 15907:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.204.121@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2568.779697] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2602.489298] Lustre: Evicted from MGS (at 192.168.204.121@tcp) after server handle changed from 0x7b8a9083d9bf88f8 to 0x7b8a9083d9bfc6fc
[ 2602.510021] Lustre: Skipped 2 previous similar messages
[ 2604.276992] Lustre: 15907:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.204.121@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2617.009371] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2639.768653] LustreError: lustre-MDT0000-mdc-ffff8f0709a47800: operation ldlm_enqueue to node 192.168.204.121@tcp failed: rc = -19
[ 2639.779719] LustreError: Skipped 5 previous similar messages
[ 2656.749676] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 192.168.204.121@tcp) was lost; in progress operations using this service will fail
[ 2656.764144] LustreError: Skipped 2 previous similar messages
[ 2665.121622] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2720.274556] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2812.686711] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 03:05:15 (1785222315)
[ 2866.366143] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 2868.090233] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3157.375318] LustreError: lustre-OST0000-osc-ffff8f0709a47800: operation ldlm_enqueue to node 192.168.204.121@tcp failed: rc = -19
[ 3157.389141] LustreError: Skipped 3 previous similar messages
[ 3157.391845] Lustre: lustre-OST0000-osc-ffff8f0709a47800: Connection to lustre-OST0000 (at 192.168.204.121@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3157.399191] Lustre: Skipped 9 previous similar messages
[ 3189.353917] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3191.000246] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3525.010616] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3527.647381] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3783.952038] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 03:21:26 (1785223286)
[ 3801.567257] Lustre: 50472:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785223289/real 1785223289]  req@ffff8f0708673b80 x1871938473742336/t0(0) o101->lustre-MDT0000-mdc-ffff8f0709a47800@192.168.204.121@tcp:12/10 lens 576/1152 e 0 to 1 dl 1785223305 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3801.598877] Lustre: 50472:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 3801.609515] Lustre: lustre-MDT0000-mdc-ffff8f0709a47800: Connection to lustre-MDT0000 (at 192.168.204.121@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3801.639324] Lustre: Skipped 1 previous similar message
[ 3801.685578] Lustre: lustre-MDT0000-mdc-ffff8f0709a47800: Connection restored to 192.168.204.121@tcp (at 192.168.204.121@tcp)
[ 3801.689597] Lustre: Skipped 18 previous similar messages
[ 3809.062448] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 03:21:51 (1785223311)
[ 3834.223942] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 03:22:17 (1785223337)
[ 3858.229984] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 03:22:41 (1785223361)
[ 3858.694590] Lustre: Mounted lustre-client
[ 3890.655624] Lustre: 2360:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785223378/real 1785223378]  req@ffff8f0708670700 x1871938473761024/t0(0) o400->lustre-MDT0000-mdc-ffff8f0709a47800@192.168.204.121@tcp:12/10 lens 224/224 e 0 to 1 dl 1785223394 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3890.657107] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 192.168.204.121@tcp) was lost; in progress operations using this service will fail
[ 3890.708280] Lustre: 2360:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 3890.719535] LustreError: Skipped 2 previous similar messages
[ 3890.730975] Lustre: Evicted from MGS (at 192.168.204.121@tcp) after server handle changed from 0x7b8a9083d9c1c921 to 0x7b8a9083d9da626c
[ 3890.744552] Lustre: Skipped 3 previous similar messages
[ 3890.750912] Lustre: MGC192.168.204.121@tcp: Connection restored to 192.168.204.121@tcp (at 192.168.204.121@tcp)
[ 3890.755826] Lustre: Skipped 2 previous similar messages
[ 3892.613822] Lustre: 15907:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.204.121@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 3914.177473] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 3915.746795] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3917.782407] Lustre: Unmounted lustre-client
[ 3923.725308] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 03:23:46 (1785223426)
[ 4045.279691] Lustre: 2361:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785223533/real 1785223533]  req@ffff8f0708799f80 x1871938473795456/t0(0) o4->lustre-OST0000-osc-ffff8f0709a47800@192.168.204.121@tcp:6/4 lens 488/448 e 0 to 1 dl 1785223549 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4045.317128] Lustre: 2361:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 61 previous similar messages
[ 4045.319247] Lustre: lustre-OST0000-osc-ffff8f0709a47800: Connection restored to 192.168.204.121@tcp (at 192.168.204.121@tcp)
[ 4045.346193] Lustre: Skipped 7 previous similar messages
[ 4217.584729] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 03:28:40 (1785223720)
[ 4265.924977] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 03:29:29 (1785223769)
[ 4268.621649] Lustre: Unmounted lustre-client
[ 4287.283823] Lustre: Mounted lustre-client
[ 4292.999573] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 03:29:56 (1785223796)
[ 4293.213856] LustreError: 56386:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4294.263931] LustreError: 56386:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4294.383171] Lustre: *** cfs_fail_loc=305, val=0***
[ 4316.076131] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 03:30:19 (1785223819)
[ 4316.363057] Lustre: Mounted lustre-client
[ 4317.906820] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4328.220358] Lustre: Unmounted lustre-client
[ 4334.503935] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 03:30:37 (1785223837)
[ 4415.735943] LustreError: lustre-MDT0000-mdc-ffff8f071081f000: operation mds_reint to node 192.168.204.121@tcp failed: rc = -19
[ 4415.744028] LustreError: Skipped 1 previous similar message
[ 4415.749135] Lustre: lustre-MDT0000-mdc-ffff8f071081f000: Connection to lustre-MDT0000 (at 192.168.204.121@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4415.760899] Lustre: Skipped 23 previous similar messages
[ 4434.410797] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 192.168.204.121@tcp) was lost; in progress operations using this service will fail
[ 4434.440984] Lustre: Evicted from MGS (at 192.168.204.121@tcp) after server handle changed from 0x7b8a9083d9da808e to 0x7b8a9083d9dd38a5
[ 4434.455095] Lustre: MGC192.168.204.121@tcp: Connection restored to 192.168.204.121@tcp (at 192.168.204.121@tcp)
[ 4434.462687] Lustre: Skipped 12 previous similar messages
[ 4568.563480] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 03:34:31 (1785224071)
[ 4572.548900] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4585.963298] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 192.168.204.121@tcp) was lost; in progress operations using this service will fail
[ 4585.988690] Lustre: Evicted from MGS (at 192.168.204.121@tcp) after server handle changed from 0x7b8a9083d9dd38a5 to 0x7b8a9083d9e1fded
[ 4596.246804] LustreError: lustre-MDT0000-mdc-ffff8f071081f000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4603.019495] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 03:35:06 (1785224106)
[ 4603.418869] Lustre: Mounted lustre-client
[ 4610.449318] LustreError: lustre-OST0000-osc-ffff8f071081f000: operation ldlm_enqueue to node 192.168.204.121@tcp failed: rc = -107
[ 4610.469447] LustreError: lustre-OST0000-osc-ffff8f071081f000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4610.494782] LustreError: 60077:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff8f071081f000: namespace resource [0x240000400:0x4a83:0x0].0x0 (ffff8f070680c900) refcount nonzero (2) after lock cleanup; forcing cleanup.
[ 4619.743346] Lustre: 2359:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785224108/real 1785224108]  req@ffff8f071029ce00 x1871938478446848/t0(0) o103->lustre-OST0000-osc-ffff8f071830c000@192.168.204.121@tcp:17/18 lens 328/224 e 0 to 1 dl 1785224124 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295
[ 4619.773380] Lustre: 2359:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 82 previous similar messages
[ 4621.659713] Lustre: Unmounted lustre-client
[ 4627.201544] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 03:35:30 (1785224130)
[ 4633.295580] LustreError: lustre-MDT0000-mdc-ffff8f071081f000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4633.314492] LustreError: 60746:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4633.320119] LustreError: 60830:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8f071081f000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff8f07200fc100) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4633.337794] LustreError: 60830:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4639.884064] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 03:35:43 (1785224143)
[ 4640.134449] LustreError: 61451:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4643.968215] LustreError: lustre-MDT0000-mdc-ffff8f071081f000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4643.979832] LustreError: 61467:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4643.997497] LustreError: 61451:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=1143
[ 4644.006517] LustreError: 61451:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting
[ 4644.032819] LustreError: 61472:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4645.086415] LustreError: 61478:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4645.097945] LustreError: 61478:0:(file.c:6154:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4647.325767] LustreError: 61501:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4647.333571] LustreError: 61501:0:(file.c:6154:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4651.772502] LustreError: 61545:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4651.777863] LustreError: 61545:0:(file.c:6154:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4659.536937] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 03:36:02 (1785224162)
[ 4691.977368] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4693.160059] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4701.882619] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 03:36:44 (1785224204)
[ 4732.416532] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4733.518862] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4740.883502] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 03:37:24 (1785224244)
[ 4798.988522] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4800.189704] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4807.211374] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 03:38:30 (1785224310)
[ 4836.699209] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4837.842333] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4842.135397] Lustre: Unmounted lustre-client
[ 4882.185517] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4884.488686] Lustre: Mounted lustre-client
[ 4889.965714] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 03:39:53 (1785224393)
[ 4891.991189] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 4893.274248] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 03:39:56 (1785224396)
[ 4912.730529] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 03:40:16 (1785224416)
[ 4913.559780] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 4914.463642] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 03:40:18 (1785224418)
[ 4915.149325] Lustre: *** cfs_fail_loc=805, val=0***
[ 4915.183268] Lustre: Mounted lustre-client
[ 4917.874229] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4935.653938] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 192.168.204.121@tcp) was lost; in progress operations using this service will fail
[ 4935.668836] Lustre: Evicted from MGS (at 192.168.204.121@tcp) after server handle changed from 0x7b8a9083d9e207c5 to 0x7b8a9083d9e20b61
[ 4939.627974] Lustre: Unmounted lustre-client
[ 4944.851227] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 03:40:48 (1785224448)
[ 4970.949101] LustreError: 2357:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff8f071029df80 x1871938478520192/t94489280516(94489280516) o101->lustre-MDT0000-mdc-ffff8f070522d800@192.168.204.121@tcp:12/10 lens 664/608 e 0 to 0 dl 1785224491 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 4974.366525] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4975.390713] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4980.973542] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 03:41:24 (1785224484)
[ 4981.800615] LustreError: lustre-OST0000-osc-ffff8f070522d800: operation ost_write to node 192.168.204.121@tcp failed: rc = -107
[ 4981.806782] LustreError: Skipped 2 previous similar messages
[ 4981.814370] LustreError: lustre-OST0000-osc-ffff8f070522d800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4981.825087] Lustre: 2358:0:(llite_lib.c:4339:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.121@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 4987.289922] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 03:41:30 (1785224490)
[ 4988.167557] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 4989.280937] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 03:41:32 (1785224492)
[ 4990.347442] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 4991.585126] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 03:41:34 (1785224494)
[ 4992.454668] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 4993.364416] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 03:41:36 (1785224496)
[ 4994.250584] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 4995.174776] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 03:41:38 (1785224498)
[ 4996.080391] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 4997.080929] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 03:41:40 (1785224500)
[ 4997.866431] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 4998.766946] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 03:41:42 (1785224502)
[ 4999.605818] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5000.578479] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 03:41:44 (1785224504)
[ 5001.403427] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5002.319892] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 03:41:45 (1785224505)
[ 5003.087369] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5003.976350] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 03:41:47 (1785224507)
[ 5004.868741] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5005.939440] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 03:41:49 (1785224509)
[ 5006.788592] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5007.658343] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 03:41:51 (1785224511)
[ 5008.930954] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5009.753553] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 03:41:53 (1785224513)
[ 5022.194025] Lustre: lustre-MDT0000-mdc-ffff8f070522d800: Connection to lustre-MDT0000 (at 192.168.204.121@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5022.207323] Lustre: Skipped 16 previous similar messages
[ 5027.922341] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 03:42:11 (1785224531)
[ 5053.009664] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 03:42:36 (1785224556)
[ 5053.282142] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5060.466143] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 03:42:44 (1785224564)
[ 5060.641768] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5067.877483] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 03:42:51 (1785224571)
[ 5068.076241] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5073.224119] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 03:42:56 (1785224576)
[ 5073.442956] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5078.473687] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 03:43:02 (1785224582)
[ 5078.646464] Lustre: *** cfs_fail_loc=510, val=3***
[ 5084.061685] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 03:43:07 (1785224587)
[ 5084.288701] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5092.107709] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 03:43:15 (1785224595)
[ 5092.318621] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5154.734367] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 03:44:18 (1785224658)
[ 5154.950171] LustreError: 82747:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5157.448424] LustreError: lustre-MDT0000-mdc-ffff8f070522d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5157.456384] LustreError: 82761:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8f070522d800: inode [0x20000a042:0x8:0x0] mdc close failed: rc = -108
[ 5157.469099] LustreError: 82761:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8f070522d800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8f070680c800) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5157.478885] Lustre: lustre-MDT0000-mdc-ffff8f070522d800: Connection restored to 192.168.204.121@tcp (at 192.168.204.121@tcp)
[ 5157.479201] LustreError: 82766:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5157.485204] Lustre: Skipped 19 previous similar messages
[ 5157.488971] LustreError: 82766:0:(file.c:6154:ll_inode_revalidate_fini()) Skipped 14 previous similar messages
[ 5159.007143] LustreError: 82747:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5161.071960] LustreError: 82777:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5163.556467] LustreError: lustre-MDT0000-mdc-ffff8f070522d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5163.566943] LustreError: 82791:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8f070522d800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8f070680c500) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5163.576256] LustreError: 82791:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 2 previous similar messages
[ 5165.137310] LustreError: 82777:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5171.633051] LustreError: lustre-MDT0000-mdc-ffff8f070522d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5171.644140] LustreError: 82819:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8f070522d800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff8f07200fcc00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5171.660409] LustreError: 82819:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5171.884216] LustreError: 82832:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5174.356173] LustreError: lustre-MDT0000-mdc-ffff8f070522d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5174.389818] LustreError: 82852:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5175.951125] LustreError: 82832:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5175.956958] LustreError: 82832:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8f070522d800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5175.966637] LustreError: 82832:0:(file.c:251:ll_close_inode_openhandle()) Skipped 4 previous similar messages
[ 5180.940356] LustreError: 82907:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5183.427292] LustreError: lustre-MDT0000-mdc-ffff8f070522d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5183.439983] LustreError: 82921:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8f070522d800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8f070680ce00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5185.015790] LustreError: 82907:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5191.434215] LustreError: lustre-MDT0000-mdc-ffff8f070522d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5191.615537] LustreError: 82963:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5194.182175] LustreError: lustre-MDT0000-mdc-ffff8f070522d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5195.680753] LustreError: 82963:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5198.274238] LustreError: lustre-MDT0000-mdc-ffff8f070522d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5199.791773] LustreError: 82991:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8f070522d800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5206.834090] LustreError: 83062:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5206.838536] LustreError: 83062:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5209.545639] LustreError: lustre-MDT0000-mdc-ffff8f070522d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5209.581653] LustreError: 83085:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5209.587810] LustreError: 83085:0:(file.c:6154:ll_inode_revalidate_fini()) Skipped 53 previous similar messages
[ 5210.903142] LustreError: 83062:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5210.912759] LustreError: 83062:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5210.917686] LustreError: 83062:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8f070522d800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5217.764798] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 03:45:21 (1785224721)
[ 5234.143181] Lustre: 83771:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785224722/real 1785224722]  req@ffff8f0718451500 x1871938478622464/t0(0) o101->lustre-MDT0000-mdc-ffff8f070522d800@192.168.204.121@tcp:12/10 lens 576/1152 e 0 to 1 dl 1785224738 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'stat.0' uid:0 gid:0 projid:0
[ 5234.165221] Lustre: 83771:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 10 previous similar messages
[ 5235.087967] LustreError: 79504:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5235.093350] LustreError: 79504:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5239.151127] LustreError: 79504:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5239.154804] LustreError: 79504:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5242.268928] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 03:45:45 (1785224745)
[ 5287.909645] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 03:46:31 (1785224791)
[ 5348.012133] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 03:47:31 (1785224851)
[ 5376.449529] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 03:47:59 (1785224879)
[ 5376.678735] Lustre: Mounted lustre-client
[ 5501.009147] Lustre: Unmounted lustre-client
[ 5511.866867] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 03:50:14 (1785225014)
[ 5512.478296] LustreError: 87387:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5516.284103] LustreError: lustre-OST0000-osc-ffff8f070522d800: operation ost_statfs to node 192.168.204.121@tcp failed: rc = -107
[ 5516.295852] LustreError: Skipped 9 previous similar messages
[ 5516.304509] LustreError: lustre-OST0000-osc-ffff8f070522d800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5516.314409] LustreError: 87474:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff8f070522d800: namespace resource [0x240000400:0x4b0d:0x0].0x0 (ffff8f071010e900) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5516.337784] LustreError: 87474:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 3 previous similar messages
[ 5516.599255] LustreError: 87387:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5516.608488] Lustre: 2360:0:(llite_lib.c:4339:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.121@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5522.877946] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 03:50:26 (1785225026)
[ 5587.745536] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 03:51:30 (1785225090)
[ 5589.534836] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 5591.465946] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 03:51:34 (1785225094)
[ 5648.351397] Lustre: lustre-MDT0000-mdc-ffff8f070522d800: Connection to lustre-MDT0000 (at 192.168.204.121@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5648.365051] Lustre: Skipped 15 previous similar messages
[ 5654.605548] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 5656.263704] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 03:52:39 (1785225159)
[ 5713.995900] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 03:53:36 (1785225216)
[ 5715.911756] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 5718.041089] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 03:53:40 (1785225220)
[ 5720.478246] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 5722.366366] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 03:53:45 (1785225225)
[ 5759.450938] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 03:54:22 (1785225262)
[ 5771.098865] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5798.884067] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 192.168.204.121@tcp) was lost; in progress operations using this service will fail
[ 5798.909967] LustreError: Skipped 2 previous similar messages
[ 5798.930937] Lustre: Evicted from MGS (at 192.168.204.121@tcp) after server handle changed from 0x7b8a9083d9e21612 to 0x7b8a9083d9e2300c
[ 5798.944941] Lustre: Skipped 2 previous similar messages
[ 5798.956716] Lustre: MGC192.168.204.121@tcp: Connection restored to 192.168.204.121@tcp (at 192.168.204.121@tcp)
[ 5798.967637] Lustre: Skipped 15 previous similar messages
[ 5812.274189] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5813.724047] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5824.540564] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 03:55:27 (1785225327)
[ 5827.870788] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 5830.197562] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 03:55:32 (1785225332)
[ 5856.224252] Lustre: 2359:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785225339/real 1785225339]  req@ffff8f071029c700 x1871938478723712/t0(0) o400->lustre-MDT0000-mdc-ffff8f070522d800@192.168.204.121@tcp:12/10 lens 224/224 e 0 to 1 dl 1785225360 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5856.259114] Lustre: 2359:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 9 previous similar messages
[ 5861.610119] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 03:56:04 (1785225364)
[ 5910.477849] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 03:56:53 (1785225413)
[ 5964.655219] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5967.259460] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6066.217462] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6067.833741] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6113.258517] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6114.771353] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6246.509513] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 04:02:28 (1785225748)
[ 6255.567564] LustreError: lustre-OST0000-osc-ffff8f070522d800: operation ost_setattr to node 192.168.204.121@tcp failed: rc = -19
[ 6255.579018] LustreError: Skipped 2 previous similar messages
[ 6255.591625] Lustre: lustre-OST0000-osc-ffff8f070522d800: Connection to lustre-OST0000 (at 192.168.204.121@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6255.610362] Lustre: Skipped 7 previous similar messages
[ 6311.334784] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6316.078863] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6552.494778] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 04:07:35 (1785226055)
[ 6735.855301] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 192.168.204.121@tcp) was lost; in progress operations using this service will fail
[ 6735.871690] LustreError: Skipped 4 previous similar messages
[ 6735.911811] Lustre: Evicted from MGS (at 192.168.204.121@tcp) after server handle changed from 0x7b8a9083d9e24a0d to 0x7b8a9083d9f0f26f
[ 6735.928762] Lustre: Skipped 4 previous similar messages
[ 6735.938586] Lustre: MGC192.168.204.121@tcp: Connection restored to 192.168.204.121@tcp (at 192.168.204.121@tcp)
[ 6735.949760] Lustre: Skipped 9 previous similar messages
[ 6761.439157] Lustre: 2361:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785226229/real 1785226229]  req@ffff8f071829f480 x1871938495232384/t0(0) o400->lustre-MDT0000-mdc-ffff8f070522d800@192.168.204.121@tcp:12/10 lens 224/224 e 0 to 1 dl 1785226265 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6761.494162] Lustre: 2361:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 18 previous similar messages
[ 6767.365339] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 04:11:10 (1785226270)
[ 6768.819646] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 6770.485634] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 04:11:13 (1785226273)
[ 6772.716261] LustreError: lustre-MDT0000-mdc-ffff8f070522d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6781.315840] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 04:11:24 (1785226284)
[ 6948.842048] LustreError: lustre-OST0000-osc-ffff8f070522d800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6950.654986] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 04:14:13 (1785226453)
[ 6975.457879] Lustre: lustre-OST0000-osc-ffff8f070522d800: Connection to lustre-OST0000 (at 192.168.204.121@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6975.480190] Lustre: Skipped 3 previous similar messages
[ 6993.711844] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 04:14:56 (1785226496)
[ 6995.122726] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 6996.686779] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 04:14:59 (1785226499)
[ 6998.211660] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 6999.892286] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 04:15:02 (1785226502)
[ 7032.800333] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 04:15:35 (1785226535)
[ 7086.665145] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 04:16:28 (1785226588)
[ 7088.605601] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7090.528968] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 04:16:33 (1785226593)
[ 7091.916664] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7093.878886] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 04:16:36 (1785226596)
[ 7109.898364] Lustre: Unmounted lustre-client
[ 7111.866124] Lustre: Mounted lustre-client
[ 7114.442423] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7146.118551] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2