[ 1836.972652] LustreError: lustre-OST0000-osc-ffff8e7e89ea3800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1836.990983] Lustre: 2364:0:(llite_lib.c:4226:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.145@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1845.431250] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 02:26:13 (1768548373)
[ 1847.693166] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1850.221647] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 02:26:17 (1768548377)
[ 1852.305813] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1854.782140] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 02:26:22 (1768548382)
[ 1859.798815] LustreError: lustre-MDT0000-mdc-ffff8e7e89ea3800: operation mds_reint to node 192.168.202.145@tcp failed: rc = -19
[ 1859.802718] LustreError: Skipped 2 previous similar messages
[ 1878.943277] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 192.168.202.145@tcp) was lost; in progress operations using this service will fail
[ 1889.217667] Lustre: Evicted from MGS (at 192.168.202.145@tcp) after server handle changed from 0x593947a19227d9c7 to 0x593947a19227f27f
[ 1994.708472] LustreError: lustre-MDT0000-mdc-ffff8e7e89ea3800: operation mds_reint to node 192.168.202.145@tcp failed: rc = -19
[ 2016.223851] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 192.168.202.145@tcp) was lost; in progress operations using this service will fail
[ 2016.255650] Lustre: Evicted from MGS (at 192.168.202.145@tcp) after server handle changed from 0x593947a19227f27f to 0x593947a19229c7be
[ 2043.918913] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 02:29:32 (1768548572)
[ 2044.358399] Lustre: *** cfs_fail_loc=305, val=0***
[ 2044.368762] Lustre: Skipped 2 previous similar messages
[ 2086.623286] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 192.168.202.145@tcp) was lost; in progress operations using this service will fail
[ 2092.011716] Lustre: lustre-MDT0000-mdc-ffff8e7e89ea3800: Connection to lustre-MDT0000 (at 192.168.202.145@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2092.030389] Lustre: Skipped 11 previous similar messages
[ 2092.181084] Lustre: lustre-MDT0000-mdc-ffff8e7e89ea3800: Connection restored to 192.168.202.145@tcp (at 192.168.202.145@tcp)
[ 2092.188818] Lustre: Skipped 12 previous similar messages
[ 2097.269622] Lustre: Evicted from MGS (at 192.168.202.145@tcp) after server handle changed from 0x593947a19229c7be to 0x593947a19229cfc8
[ 2104.249062] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2106.060584] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2115.868912] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 02:30:43 (1768548643)
[ 2133.504315] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 192.168.202.145@tcp) was lost; in progress operations using this service will fail
[ 2133.524864] Lustre: Evicted from MGS (at 192.168.202.145@tcp) after server handle changed from 0x593947a19229cfc8 to 0x593947a19229d268
[ 2133.657530] LustreError: lustre-MDT0000-mdc-ffff8e7e89ea3800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2157.680490] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 02:31:25 (1768548685)
[ 2174.459294] LustreError: lustre-OST0000-osc-ffff8e7e89ea3800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2195.118231] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 02:32:03 (1768548723)
[ 2209.742061] LustreError: lustre-MDT0000-mdc-ffff8e7e89ea3800: operation ldlm_enqueue to node 192.168.202.145@tcp failed: rc = -19
[ 2225.635810] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 192.168.202.145@tcp) was lost; in progress operations using this service will fail
[ 2235.946466] Lustre: Evicted from MGS (at 192.168.202.145@tcp) after server handle changed from 0x593947a19229d268 to 0x593947a1922a1f30
[ 2266.597584] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2269.702868] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2352.415857] Lustre: 2362:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768548866/real 1768548866]  req@ffff8e7e8a15c700 x1854455556872192/t0(0) o400->MGC192.168.202.145@tcp@192.168.202.145@tcp:26/25 lens 224/224 e 0 to 1 dl 1768548882 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2352.477885] Lustre: 2362:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 14 previous similar messages
[ 2352.491285] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 192.168.202.145@tcp) was lost; in progress operations using this service will fail
[ 2362.671633] Lustre: Evicted from MGS (at 192.168.202.145@tcp) after server handle changed from 0x593947a1922a1f30 to 0x593947a1922bbb5e
[ 2377.071156] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2379.965672] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2465.247846] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 192.168.202.145@tcp) was lost; in progress operations using this service will fail
[ 2474.427515] Lustre: Evicted from MGS (at 192.168.202.145@tcp) after server handle changed from 0x593947a1922bbb5e to 0x593947a1922d2aad
[ 2491.426206] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2494.161417] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2525.239991] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 02:37:33 (1768549053)
[ 2530.511459] LustreError: lustre-MDT0000-mdc-ffff8e7e89ea3800: operation ldlm_enqueue to node 192.168.202.145@tcp failed: rc = -19
[ 2530.523354] LustreError: Skipped 3 previous similar messages
[ 2550.256821] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 192.168.202.145@tcp) was lost; in progress operations using this service will fail
[ 2550.272946] Lustre: Evicted from MGS (at 192.168.202.145@tcp) after server handle changed from 0x593947a1922d2aad to 0x593947a1922df4cd
[ 2568.881830] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2609.800529] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2655.148797] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2685.407753] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 192.168.202.145@tcp) was lost; in progress operations using this service will fail
[ 2685.421560] LustreError: Skipped 2 previous similar messages
[ 2695.593590] Lustre: Evicted from MGS (at 192.168.202.145@tcp) after server handle changed from 0x593947a1922e3237 to 0x593947a1922e85b9
[ 2695.614249] Lustre: Skipped 2 previous similar messages
[ 2695.627245] Lustre: MGC192.168.202.145@tcp: Connection restored to 192.168.202.145@tcp (at 192.168.202.145@tcp)
[ 2695.640039] Lustre: Skipped 16 previous similar messages
[ 2702.095704] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2727.056563] Lustre: lustre-MDT0000-mdc-ffff8e7e89ea3800: Connection to lustre-MDT0000 (at 192.168.202.145@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2727.073518] Lustre: Skipped 9 previous similar messages
[ 2764.708320] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2836.287737] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2937.744607] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 02:44:26 (1768549466)
[ 2999.560614] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3002.663343] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3284.911811] LustreError: lustre-OST0000-osc-ffff8e7e89ea3800: operation ldlm_enqueue to node 192.168.202.145@tcp failed: rc = -107
[ 3284.928814] LustreError: Skipped 8 previous similar messages
[ 3327.394820] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3330.463394] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3616.635083] Lustre: lustre-OST0000-osc-ffff8e7e89ea3800: Connection to lustre-OST0000 (at 192.168.202.145@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3616.655203] Lustre: Skipped 4 previous similar messages
[ 3664.978559] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3668.155722] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3913.602644] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 03:00:41 (1768550441)
[ 3931.615914] Lustre: 50218:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768550445/real 1768550445]  req@ffff8e7e90c2f100 x1854455576231552/t0(0) o101->lustre-MDT0000-mdc-ffff8e7e89ea3800@192.168.202.145@tcp:12/10 lens 576/1152 e 0 to 1 dl 1768550461 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3931.667625] Lustre: 50218:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
[ 3931.698389] Lustre: lustre-MDT0000-mdc-ffff8e7e89ea3800: Connection restored to 192.168.202.145@tcp (at 192.168.202.145@tcp)
[ 3931.704454] Lustre: Skipped 8 previous similar messages
[ 3940.935446] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 03:01:08 (1768550468)
[ 3966.951579] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 03:01:35 (1768550495)
[ 3993.282541] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 03:02:01 (1768550521)
[ 3993.979068] Lustre: Mounted lustre-client
[ 4025.823604] Lustre: 2363:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768550539/real 1768550539]  req@ffff8e7e90c78000 x1854455576248448/t0(0) o400->lustre-MDT0000-mdc-ffff8e7e89ea3800@192.168.202.145@tcp:12/10 lens 224/224 e 0 to 1 dl 1768550555 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4025.855119] Lustre: 2363:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 4025.864724] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 192.168.202.145@tcp) was lost; in progress operations using this service will fail
[ 4025.876401] LustreError: Skipped 3 previous similar messages
[ 4036.090124] Lustre: Evicted from MGS (at 192.168.202.145@tcp) after server handle changed from 0x593947a1923051f9 to 0x593947a192425f57
[ 4036.102172] Lustre: Skipped 3 previous similar messages
[ 4036.109948] Lustre: MGC192.168.202.145@tcp: Connection restored to 192.168.202.145@tcp (at 192.168.202.145@tcp)
[ 4036.132541] Lustre: Skipped 2 previous similar messages
[ 4049.502732] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4051.595837] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4053.832964] LustreError: 52986:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff8e7e84e92000: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 4053.848502] LustreError: 52986:0:(lov_obd.c:783:lov_cleanup()) Skipped 1 previous similar message
[ 4053.861094] LustreError: 52986:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 4053.866809] LustreError: 52986:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 4053.915894] Lustre: Unmounted lustre-client
[ 4061.298649] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 03:03:09 (1768550589)
[ 4186.079235] Lustre: 2363:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768550699/real 1768550699]  req@ffff8e7e863f6300 x1854455576281472/t0(0) o4->lustre-OST0000-osc-ffff8e7e89ea3800@192.168.202.145@tcp:6/4 lens 488/448 e 0 to 1 dl 1768550715 ref 2 fl Rpc:XQr/602/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4186.116319] Lustre: 2363:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 56 previous similar messages
[ 4186.117592] Lustre: lustre-OST0000-osc-ffff8e7e89ea3800: Connection restored to 192.168.202.145@tcp (at 192.168.202.145@tcp)
[ 4186.135423] Lustre: Skipped 7 previous similar messages
[ 4217.823294] Lustre: lustre-OST0000-osc-ffff8e7e89ea3800: Connection to lustre-OST0000 (at 192.168.202.145@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4217.831609] Lustre: Skipped 13 previous similar messages
[ 4360.291784] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 03:08:08 (1768550888)
[ 4410.231066] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 03:08:58 (1768550938)
[ 4414.740928] LustreError: 55144:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff8e7e89ea3800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 4414.750349] LustreError: 55144:0:(lov_obd.c:783:lov_cleanup()) Skipped 1 previous similar message
[ 4414.785197] LustreError: 55144:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup
[ 4414.791296] LustreError: 55144:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 4414.833656] Lustre: Unmounted lustre-client
[ 4440.691686] Lustre: Mounted lustre-client
[ 4449.535787] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 03:09:36 (1768550976)
[ 4450.008427] LustreError: 56128:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4451.065290] LustreError: 56128:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4451.246635] Lustre: *** cfs_fail_loc=305, val=0***
[ 4474.665338] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 03:10:02 (1768551002)
[ 4475.456667] Lustre: Mounted lustre-client
[ 4477.485853] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4487.807325] LustreError: 56805:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 4487.818700] LustreError: 56805:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages
[ 4487.863431] Lustre: Unmounted lustre-client
[ 4495.072579] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 03:10:23 (1768551023)
[ 4584.976992] LustreError: lustre-MDT0000-mdc-ffff8e7e88d69800: operation mds_reint to node 192.168.202.145@tcp failed: rc = -19
[ 4584.990139] LustreError: Skipped 1 previous similar message
[ 4604.383172] Lustre: 2361:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768551118/real 1768551118]  req@ffff8e7e89b88a80 x1854455578950144/t0(0) o400->MGC192.168.202.145@tcp@192.168.202.145@tcp:26/25 lens 224/224 e 0 to 1 dl 1768551134 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4604.419433] Lustre: 2361:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 74 previous similar messages
[ 4604.425063] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 192.168.202.145@tcp) was lost; in progress operations using this service will fail
[ 4614.630214] Lustre: Evicted from MGS (at 192.168.202.145@tcp) after server handle changed from 0x593947a19242770c to 0x593947a192452d24
[ 4614.635601] Lustre: MGC192.168.202.145@tcp: Connection restored to 192.168.202.145@tcp (at 192.168.202.145@tcp)
[ 4614.640314] Lustre: Skipped 12 previous similar messages
[ 4785.496181] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 03:15:13 (1768551313)
[ 4790.855271] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4808.216249] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 192.168.202.145@tcp) was lost; in progress operations using this service will fail
[ 4808.245461] Lustre: Evicted from MGS (at 192.168.202.145@tcp) after server handle changed from 0x593947a192452d24 to 0x593947a19249f5de
[ 4808.321141] LustreError: lustre-MDT0000-mdc-ffff8e7e88d69800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4824.778345] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 03:15:52 (1768551352)
[ 4825.361921] Lustre: Mounted lustre-client
[ 4832.690264] LustreError: lustre-OST0000-osc-ffff8e7e88d69800: operation ldlm_enqueue to node 192.168.202.145@tcp failed: rc = -107
[ 4832.698221] Lustre: lustre-OST0000-osc-ffff8e7e88d69800: Connection to lustre-OST0000 (at 192.168.202.145@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4832.717138] Lustre: Skipped 12 previous similar messages
[ 4832.731871] LustreError: lustre-OST0000-osc-ffff8e7e88d69800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4832.745563] LustreError: 59841:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-OST0000-osc-ffff8e7e88d69800: namespace resource [0x240000400:0x38c3:0x0].0x0 (ffff8e7e846ec900) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4845.800592] LustreError: 59871:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff8e7e8b03c800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 4845.813027] LustreError: 59871:0:(lov_obd.c:783:lov_cleanup()) Skipped 1 previous similar message
[ 4845.823737] LustreError: 59871:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 4845.832313] LustreError: 59871:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 4845.895271] Lustre: Unmounted lustre-client
[ 4853.964627] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 03:16:21 (1768551381)
[ 4861.109547] LustreError: lustre-MDT0000-mdc-ffff8e7e88d69800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4861.149339] LustreError: 60512:0:(file.c:6123:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4873.759870] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 03:16:41 (1768551401)
[ 4873.998185] LustreError: 61213:0:(recover.c:330:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4878.181471] LustreError: lustre-MDT0000-mdc-ffff8e7e88d69800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4878.197930] LustreError: 61234:0:(import.c:291:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4878.205110] LustreError: 61213:0:(recover.c:330:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=800
[ 4878.222164] LustreError: 61213:0:(import.c:716:ptlrpc_connect_import_locked()) already connecting
[ 4878.278858] LustreError: 61239:0:(file.c:6123:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4879.419706] LustreError: 61246:0:(file.c:6123:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4879.436463] LustreError: 61246:0:(file.c:6123:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4881.852473] LustreError: 61268:0:(file.c:6123:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4881.873268] LustreError: 61268:0:(file.c:6123:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4886.596633] LustreError: 61313:0:(file.c:6123:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4886.612437] LustreError: 61313:0:(file.c:6123:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4897.177312] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 03:17:05 (1768551425)
[ 4937.868831] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4939.457723] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4949.740671] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 03:17:58 (1768551478)
[ 4994.145663] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4996.472074] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5009.549767] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 03:18:57 (1768551537)
[ 5077.610716] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5079.386303] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5088.324743] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 03:20:16 (1768551616)
[ 5126.471229] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5128.151812] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5134.261694] LustreError: 66890:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff8e7e88d69800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 5134.278863] LustreError: 66890:0:(lov_obd.c:783:lov_cleanup()) Skipped 1 previous similar message
[ 5134.307697] LustreError: 66890:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup
[ 5134.312294] LustreError: 66890:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 5134.355490] Lustre: Unmounted lustre-client
[ 5192.352754] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5195.526698] Lustre: Mounted lustre-client
[ 5202.701909] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 03:22:10 (1768551730)
[ 5205.541592] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5206.993655] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 03:22:15 (1768551735)
[ 5240.482818] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 03:22:48 (1768551768)
[ 5242.292036] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5243.989978] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 03:22:52 (1768551772)
[ 5245.662640] Lustre: *** cfs_fail_loc=805, val=0***
[ 5245.737632] Lustre: Mounted lustre-client
[ 5251.175110] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5271.520707] Lustre: 2363:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768551785/real 1768551785]  req@ffff8e7e846b8380 x1854455581005312/t0(0) o400->lustre-MDT0000-mdc-ffff8e7e90c39000@192.168.202.145@tcp:12/10 lens 224/224 e 0 to 1 dl 1768551801 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5271.548880] Lustre: 2363:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5271.555771] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 192.168.202.145@tcp) was lost; in progress operations using this service will fail
[ 5281.769845] Lustre: Evicted from MGS (at 192.168.202.145@tcp) after server handle changed from 0x593947a19249ffcb to 0x593947a1924a0367
[ 5281.785791] Lustre: MGC192.168.202.145@tcp: Connection restored to 192.168.202.145@tcp (at 192.168.202.145@tcp)
[ 5281.791887] LustreError: 2360:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8e7e88b0e800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff8e7e84690c00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5281.796486] Lustre: Skipped 12 previous similar messages
[ 5281.824377] LustreError: 2360:0:(ldlm_resource.c:981:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5283.867279] LustreError: 70235:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff8e7e88b0e800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 5283.870930] LustreError: 70235:0:(lov_obd.c:783:lov_cleanup()) Skipped 1 previous similar message
[ 5283.885914] LustreError: 70235:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 5283.888326] LustreError: 70235:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages
[ 5283.905181] Lustre: Unmounted lustre-client
[ 5294.739313] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 03:23:42 (1768551822)
[ 5321.796964] LustreError: 2360:0:(client.c:3418:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff8e7e846b8e00 x1854455581016320/t94489280516(94489280516) o101->lustre-MDT0000-mdc-ffff8e7e90c39000@192.168.202.145@tcp:12/10 lens 664/608 e 0 to 0 dl 1768551867 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5333.049631] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5334.898535] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5343.296955] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 03:24:31 (1768551871)
[ 5344.809209] LustreError: lustre-OST0000-osc-ffff8e7e90c39000: operation ost_write to node 192.168.202.145@tcp failed: rc = -107
[ 5344.812445] LustreError: Skipped 2 previous similar messages
[ 5344.845562] LustreError: lustre-OST0000-osc-ffff8e7e90c39000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5344.856551] Lustre: 2363:0:(llite_lib.c:4226:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.145@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5353.161677] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 03:24:41 (1768551881)
[ 5354.620724] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5356.580950] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 03:24:44 (1768551884)
[ 5358.265819] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5360.081083] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 03:24:48 (1768551888)
[ 5361.359159] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5363.019618] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 03:24:51 (1768551891)
[ 5364.779730] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5366.382781] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 03:24:54 (1768551894)
[ 5367.740412] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5369.184779] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 03:24:57 (1768551897)
[ 5370.404481] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5372.651219] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 03:25:00 (1768551900)
[ 5373.963452] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5376.052302] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 03:25:04 (1768551904)
[ 5377.614509] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5379.072264] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 03:25:07 (1768551907)
[ 5380.419682] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5382.523474] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 03:25:10 (1768551910)
[ 5384.589560] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5386.520330] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 03:25:14 (1768551914)
[ 5388.009514] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5389.791601] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 03:25:18 (1768551918)
[ 5392.348980] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5393.804874] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 03:25:22 (1768551922)
[ 5430.677093] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 03:25:58 (1768551958)
[ 5460.015195] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 03:26:28 (1768551988)
[ 5460.603524] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5471.686223] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 03:26:39 (1768551999)
[ 5472.290154] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5484.420779] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 03:26:52 (1768552012)
[ 5484.622858] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5493.891979] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 03:27:02 (1768552022)
[ 5494.310889] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5503.213518] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 03:27:11 (1768552031)
[ 5503.680340] Lustre: *** cfs_fail_loc=510, val=3***
[ 5513.275244] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 03:27:21 (1768552041)
[ 5514.043893] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5524.891904] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 03:27:33 (1768552053)
[ 5525.575149] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5585.313505] Lustre: lustre-OST0000-osc-ffff8e7e90c39000: Connection to lustre-OST0000 (at 192.168.202.145@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5585.333986] Lustre: Skipped 15 previous similar messages
[ 5592.024209] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 03:28:40 (1768552120)
[ 5592.618291] LustreError: 82427:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5595.671475] LustreError: lustre-MDT0000-mdc-ffff8e7e90c39000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5595.687107] LustreError: 82442:0:(file.c:249:ll_close_inode_openhandle()) lustre-clilmv-ffff8e7e90c39000: inode [0x20000a042:0xb:0x0] mdc close failed: rc = -108
[ 5595.719435] LustreError: 82442:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8e7e90c39000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8e7e846ec200) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5595.754029] LustreError: 82442:0:(ldlm_resource.c:981:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5596.695147] LustreError: 82427:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5598.885561] LustreError: 82457:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5602.335951] LustreError: lustre-MDT0000-mdc-ffff8e7e90c39000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5602.366219] LustreError: 82472:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8e7e90c39000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8e7e8a170700) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5602.410674] LustreError: 82472:0:(ldlm_resource.c:981:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5602.983143] LustreError: 82457:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5610.045120] LustreError: lustre-MDT0000-mdc-ffff8e7e90c39000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5610.548497] LustreError: 82513:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5613.914753] LustreError: lustre-MDT0000-mdc-ffff8e7e90c39000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5613.974851] LustreError: 82533:0:(file.c:6123:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5613.980089] LustreError: 82533:0:(file.c:6123:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5614.615246] LustreError: 82513:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5614.628318] LustreError: 82513:0:(file.c:249:ll_close_inode_openhandle()) lustre-clilmv-ffff8e7e90c39000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5615.062700] LustreError: 82539:0:(file.c:6123:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5615.072790] LustreError: 82539:0:(file.c:6123:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5617.410910] LustreError: 82561:0:(file.c:6123:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5617.418266] LustreError: 82561:0:(file.c:6123:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5621.330970] LustreError: 82588:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5624.667618] LustreError: lustre-MDT0000-mdc-ffff8e7e90c39000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5624.680479] LustreError: 82603:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8e7e90c39000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8e7e845e9b00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5624.700850] LustreError: 82603:0:(ldlm_resource.c:981:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5625.399493] LustreError: 82588:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5632.491607] LustreError: lustre-MDT0000-mdc-ffff8e7e90c39000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5633.093368] LustreError: 82645:0:(ldlm_flock.c:856:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5636.457227] LustreError: lustre-MDT0000-mdc-ffff8e7e90c39000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5636.496170] LustreError: 82660:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8e7e90c39000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8e7e845e9300) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5636.585079] LustreError: 82660:0:(ldlm_resource.c:981:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5637.103118] LustreError: 82645:0:(ldlm_flock.c:856:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5640.486922] LustreError: lustre-MDT0000-mdc-ffff8e7e90c39000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5640.596723] LustreError: 82693:0:(file.c:6123:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5640.619107] LustreError: 82693:0:(file.c:6123:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5641.296891] LustreError: 82673:0:(file.c:249:ll_close_inode_openhandle()) lustre-clilmv-ffff8e7e90c39000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5649.835351] LustreError: 82745:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5649.846191] LustreError: 82745:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5653.548306] LustreError: lustre-MDT0000-mdc-ffff8e7e90c39000: operation ldlm_enqueue to node 192.168.202.145@tcp failed: rc = -107
[ 5653.559311] LustreError: Skipped 8 previous similar messages
[ 5653.579925] LustreError: lustre-MDT0000-mdc-ffff8e7e90c39000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5653.720405] LustreError: 82767:0:(file.c:6123:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5653.729954] LustreError: 82767:0:(file.c:6123:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5653.928266] LustreError: 82745:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5653.935417] LustreError: 82745:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5653.943404] LustreError: 82745:0:(file.c:249:ll_close_inode_openhandle()) lustre-clilmv-ffff8e7e90c39000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5666.204517] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 03:29:54 (1768552194)
[ 5686.308626] LustreError: 67721:0:(ldlm_lockd.c:2882:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5686.317096] LustreError: 67721:0:(ldlm_lockd.c:2882:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5690.387156] LustreError: 67721:0:(ldlm_lockd.c:2882:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5690.394083] LustreError: 67721:0:(ldlm_lockd.c:2882:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5698.801683] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 03:30:26 (1768552226)
[ 5750.055991] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 03:31:18 (1768552278)
[ 5815.641457] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 03:32:23 (1768552343)
[ 5852.662288] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 03:33:00 (1768552380)
[ 5853.428672] Lustre: Mounted lustre-client
[ 5977.839811] LustreError: 86460:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff8e7e87315800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 5977.854761] LustreError: 86460:0:(lov_obd.c:783:lov_cleanup()) Skipped 1 previous similar message
[ 5977.864628] LustreError: 86460:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 5977.871853] LustreError: 86460:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 5977.914874] Lustre: Unmounted lustre-client
[ 5985.264934] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 03:35:13 (1768552513)
[ 5986.082873] LustreError: 87076:0:(osc_request.c:2944:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 4000ms
[ 5989.705088] LustreError: lustre-OST0000-osc-ffff8e7e90c39000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5989.713677] LustreError: 87163:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-OST0000-osc-ffff8e7e90c39000: namespace resource [0x240000400:0x394d:0x0].0x0 (ffff8e7e8a170d00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5989.728511] LustreError: 87163:0:(ldlm_resource.c:981:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5990.151533] LustreError: 87076:0:(osc_request.c:2944:osc_build_rpc()) cfs_fail_timeout id 414 awake
[ 5990.163760] Lustre: 2362:0:(llite_lib.c:4226:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.145@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5990.184262] Lustre: lustre-OST0000-osc-ffff8e7e90c39000: Connection restored to 192.168.202.145@tcp (at 192.168.202.145@tcp)
[ 5990.193085] Lustre: Skipped 18 previous similar messages
[ 5997.488944] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 03:35:25 (1768552525)
[ 6056.927279] Lustre: 87743:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768552530/real 1768552530]  req@ffff8e7e89a2fb80 x1854455581188352/t0(0) o101->lustre-MDT0000-mdc-ffff8e7e90c39000@192.168.202.145@tcp:12/10 lens 328/344 e 0 to 1 dl 1768552586 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 6056.958114] Lustre: 87743:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 21 previous similar messages
[ 6065.663060] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 03:36:33 (1768552593)
[ 6067.537799] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 6069.946189] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 03:36:37 (1768552597)
[ 6136.028912] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6138.508503] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 03:37:46 (1768552666)
[ 6197.216293] Lustre: lustre-MDT0000-mdc-ffff8e7e90c39000: Connection to lustre-MDT0000 (at 192.168.202.145@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6197.246894] Lustre: Skipped 15 previous similar messages
[ 6198.908152] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 03:38:47 (1768552727)
[ 6201.486663] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6203.311570] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 03:38:51 (1768552731)
[ 6204.807857] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6207.013561] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 03:38:54 (1768552734)
[ 6249.451519] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 03:39:37 (1768552777)
[ 6262.884507] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6289.375295] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 192.168.202.145@tcp) was lost; in progress operations using this service will fail
[ 6289.386672] LustreError: Skipped 2 previous similar messages
[ 6299.714021] Lustre: Evicted from MGS (at 192.168.202.145@tcp) after server handle changed from 0x593947a1924a0e03 to 0x593947a1924a28ba
[ 6299.733055] Lustre: Skipped 2 previous similar messages
[ 6303.767946] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6306.756389] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6322.348819] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 03:40:50 (1768552850)
[ 6325.300136] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6327.559589] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 03:40:55 (1768552855)
[ 6340.594948] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 192.168.202.145@tcp) was lost; in progress operations using this service will fail
[ 6340.631648] Lustre: Evicted from MGS (at 192.168.202.145@tcp) after server handle changed from 0x593947a1924a28ba to 0x593947a1924a2a3b
[ 6357.606319] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 03:41:25 (1768552885)
[ 6418.149759] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 03:42:25 (1768552945)
[ 6425.154600] LustreError: lustre-OST0000-osc-ffff8e7e90c39000: operation ost_setattr to node 192.168.202.145@tcp failed: rc = -19
[ 6425.165470] LustreError: Skipped 3 previous similar messages
[ 6475.663217] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 6479.086328] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6565.855425] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 192.168.202.145@tcp) was lost; in progress operations using this service will fail
[ 6565.873479] LustreError: Skipped 1 previous similar message
[ 6576.124107] Lustre: Evicted from MGS (at 192.168.202.145@tcp) after server handle changed from 0x593947a1924a2cc6 to 0x593947a1924a3b05
[ 6576.142200] Lustre: Skipped 1 previous similar message
[ 6590.648252] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6592.777061] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6622.222942] Lustre: MGC192.168.202.145@tcp: Connection restored to 192.168.202.145@tcp (at 192.168.202.145@tcp)
[ 6622.240950] Lustre: Skipped 11 previous similar messages
[ 6640.778311] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6642.584961] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6762.237735] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 03:48:10 (1768553290)
[ 6829.818923] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 6835.772568] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 7082.782842] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 03:53:30 (1768553610)
[ 7253.992733] Lustre: lustre-MDT0000-mdc-ffff8e7e90c39000: Connection to lustre-MDT0000 (at 192.168.202.145@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7254.008827] Lustre: Skipped 7 previous similar messages
[ 7254.016362] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 192.168.202.145@tcp) was lost; in progress operations using this service will fail
[ 7254.030390] LustreError: Skipped 1 previous similar message
[ 7254.052324] Lustre: Evicted from MGS (at 192.168.202.145@tcp) after server handle changed from 0x593947a1924a3f0a to 0x593947a19259e2ee
[ 7254.068738] Lustre: Skipped 1 previous similar message
[ 7254.079970] Lustre: MGC192.168.202.145@tcp: Connection restored to 192.168.202.145@tcp (at 192.168.202.145@tcp)
[ 7254.095108] Lustre: Skipped 1 previous similar message
[ 7278.559515] Lustre: 2362:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768553768/real 1768553768]  req@ffff8e7e8946a300 x1854455598921344/t0(0) o400->lustre-MDT0000-mdc-ffff8e7e90c39000@192.168.202.145@tcp:12/10 lens 224/224 e 0 to 1 dl 1768553808 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7278.596672] Lustre: 2362:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 21 previous similar messages
[ 7286.521759] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 03:56:54 (1768553814)
[ 7288.735811] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7290.374898] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 03:56:58 (1768553818)
[ 7291.851068] LustreError: lustre-MDT0000-mdc-ffff8e7e90c39000: operation ldlm_enqueue to node 192.168.202.145@tcp failed: rc = -107
[ 7291.864295] LustreError: Skipped 425 previous similar messages
[ 7291.897750] LustreError: lustre-MDT0000-mdc-ffff8e7e90c39000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7301.275487] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 03:57:09 (1768553829)
[ 7468.549969] LustreError: lustre-OST0000-osc-ffff8e7e90c39000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7471.105975] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 03:59:58 (1768553998)
[ 7496.159190] Lustre: 2362:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768554005/real 1768554005]  req@ffff8e7eb3892d80 x1854455598942464/t0(0) o4->lustre-OST0000-osc-ffff8e7e90c39000@192.168.202.145@tcp:6/4 lens 4584/448 e 0 to 1 dl 1768554025 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7496.185747] Lustre: 2362:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 7515.174763] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 04:00:43 (1768554043)
[ 7517.258470] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7519.527296] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 04:00:47 (1768554047)
[ 7521.329890] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7523.387110] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 04:00:51 (1768554051)
[ 7558.427223] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 04:01:26 (1768554086)
[ 7598.571449] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 192.168.202.145@tcp) was lost; in progress operations using this service will fail
[ 7598.587882] Lustre: Evicted from MGS (at 192.168.202.145@tcp) after server handle changed from 0x593947a19259e2ee to 0x593947a1925a089c
[ 7598.595295] Lustre: Skipped 1 previous similar message
[ 7612.305168] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 04:02:20 (1768554140)
[ 7613.822322] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7615.887743] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 04:02:24 (1768554144)
[ 7617.638422] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7619.826921] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 04:02:27 (1768554147)
[ 7631.198728] LustreError: 106001:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff8e7e90c39000: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 7631.209377] LustreError: 106001:0:(lov_obd.c:783:lov_cleanup()) Skipped 1 previous similar message
[ 7631.226708] LustreError: 106001:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup
[ 7631.231708] LustreError: 106001:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 7631.339864] Lustre: Unmounted lustre-client
[ 7633.845802] Lustre: Mounted lustre-client
[ 7636.914331] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7654.369753] Lustre: 2363:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768554168/real 1768554168]  req@ffff8e7e8a1db100 x1854455599128960/t0(0) o400->MGC192.168.202.145@tcp@192.168.202.145@tcp:26/25 lens 224/224 e 0 to 1 dl 1768554184 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7654.409428] Lustre: 2363:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 7667.561262] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: total 1813