[ 1883.526560] LustreError: lustre-OST0000-osc-ffff9e6dd2328800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1883.550200] Lustre: 2347:0:(llite_lib.c:4202:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.153@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1890.893783] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 00:12:55 (1783138375)
[ 1893.549353] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1895.562508] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 00:13:00 (1783138380)
[ 1897.530810] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1899.711711] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 00:13:04 (1783138384)
[ 1921.313194] LustreError: MGC192.168.201.153@tcp: Connection to MGS (at 192.168.201.153@tcp) was lost; in progress operations using this service will fail
[ 1931.575556] Lustre: Evicted from MGS (at 192.168.201.153@tcp) after server handle changed from 0x604be086e4fea049 to 0x604be086e4febc96
[ 2025.302368] LustreError: lustre-MDT0000-mdc-ffff9e6dd2328800: operation ldlm_enqueue to node 192.168.201.153@tcp failed: rc = -19
[ 2025.315927] LustreError: Skipped 3 previous similar messages
[ 2043.367560] LustreError: MGC192.168.201.153@tcp: Connection to MGS (at 192.168.201.153@tcp) was lost; in progress operations using this service will fail
[ 2053.599078] Lustre: Evicted from MGS (at 192.168.201.153@tcp) after server handle changed from 0x604be086e4febc96 to 0x604be086e5005281
[ 2069.262300] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 00:15:54 (1783138554)
[ 2069.657631] Lustre: *** cfs_fail_loc=305, val=0***
[ 2069.661562] Lustre: Skipped 4 previous similar messages
[ 2089.527480] LustreError: lustre-OST0001-osc-ffff9e6dd2328800: operation ost_connect to node 192.168.201.153@tcp failed: rc = -75
[ 2115.042683] LustreError: MGC192.168.201.153@tcp: Connection to MGS (at 192.168.201.153@tcp) was lost; in progress operations using this service will fail
[ 2115.101150] Lustre: Evicted from MGS (at 192.168.201.153@tcp) after server handle changed from 0x604be086e5005281 to 0x604be086e5005686
[ 2116.353087] Lustre: 15884:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.201.153@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2140.148376] Lustre: DEBUG MARKER: oleg153-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2142.270911] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2151.636280] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 00:17:16 (1783138636)
[ 2166.259762] LustreError: MGC192.168.201.153@tcp: Connection to MGS (at 192.168.201.153@tcp) was lost; in progress operations using this service will fail
[ 2166.284130] Lustre: Evicted from MGS (at 192.168.201.153@tcp) after server handle changed from 0x604be086e5005686 to 0x604be086e5005c7c
[ 2166.359232] LustreError: lustre-MDT0000-mdc-ffff9e6dd2328800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2188.146696] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 00:17:52 (1783138672)
[ 2201.362330] LustreError: lustre-OST0000-osc-ffff9e6dd2328800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2221.620627] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 00:18:26 (1783138706)
[ 2235.322066] LustreError: lustre-MDT0000-mdc-ffff9e6dd2328800: operation mds_reint to node 192.168.201.153@tcp failed: rc = -19
[ 2253.297476] LustreError: MGC192.168.201.153@tcp: Connection to MGS (at 192.168.201.153@tcp) was lost; in progress operations using this service will fail
[ 2253.394447] Lustre: Evicted from MGS (at 192.168.201.153@tcp) after server handle changed from 0x604be086e5005c7c to 0x604be086e500a7bc
[ 2270.361525] Lustre: DEBUG MARKER: oleg153-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2273.378963] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2356.259534] Lustre: 2350:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783138826/real 1783138826]  req@ffff9e6dc59ba300 x1869754195737472/t0(0) o400->MGC192.168.201.153@tcp@192.168.201.153@tcp:26/25 lens 224/224 e 0 to 1 dl 1783138842 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2356.285968] Lustre: 2350:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 13 previous similar messages
[ 2356.294940] LustreError: MGC192.168.201.153@tcp: Connection to MGS (at 192.168.201.153@tcp) was lost; in progress operations using this service will fail
[ 2366.519661] Lustre: Evicted from MGS (at 192.168.201.153@tcp) after server handle changed from 0x604be086e500a7bc to 0x604be086e5027ab6
[ 2384.805943] Lustre: DEBUG MARKER: oleg153-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2388.120460] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2454.358255] Lustre: lustre-MDT0000-mdc-ffff9e6dd2328800: Connection to lustre-MDT0000 (at 192.168.201.153@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2454.375827] Lustre: Skipped 9 previous similar messages
[ 2473.442886] LustreError: MGC192.168.201.153@tcp: Connection to MGS (at 192.168.201.153@tcp) was lost; in progress operations using this service will fail
[ 2473.481515] Lustre: Evicted from MGS (at 192.168.201.153@tcp) after server handle changed from 0x604be086e5027ab6 to 0x604be086e50431a2
[ 2473.490080] Lustre: MGC192.168.201.153@tcp: Connection restored to 192.168.201.153@tcp (at 192.168.201.153@tcp)
[ 2473.509661] Lustre: Skipped 15 previous similar messages
[ 2474.757237] Lustre: 15884:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.201.153@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2502.127037] Lustre: DEBUG MARKER: oleg153-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2506.155231] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2537.392465] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 00:23:42 (1783139022)
[ 2542.124343] LustreError: lustre-MDT0000-mdc-ffff9e6dd2328800: operation mds_reint to node 192.168.201.153@tcp failed: rc = -19
[ 2542.129411] LustreError: Skipped 6 previous similar messages
[ 2559.455266] LustreError: MGC192.168.201.153@tcp: Connection to MGS (at 192.168.201.153@tcp) was lost; in progress operations using this service will fail
[ 2569.655037] Lustre: Evicted from MGS (at 192.168.201.153@tcp) after server handle changed from 0x604be086e50431a2 to 0x604be086e504ec8e
[ 2579.152135] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2605.046555] Lustre: 15884:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.201.153@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2621.478068] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2649.784503] Lustre: 15884:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.201.153@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2667.367553] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2698.721703] LustreError: MGC192.168.201.153@tcp: Connection to MGS (at 192.168.201.153@tcp) was lost; in progress operations using this service will fail
[ 2698.733373] LustreError: Skipped 2 previous similar messages
[ 2708.982792] Lustre: Evicted from MGS (at 192.168.201.153@tcp) after server handle changed from 0x604be086e5050a47 to 0x604be086e505849d
[ 2709.001954] Lustre: Skipped 2 previous similar messages
[ 2709.258104] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2774.355420] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2832.796401] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2920.722446] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 00:30:05 (1783139405)
[ 2978.293348] Lustre: DEBUG MARKER: oleg153-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2981.379691] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3265.518155] LustreError: lustre-OST0000-osc-ffff9e6dd2328800: operation ldlm_enqueue to node 192.168.201.153@tcp failed: rc = -19
[ 3265.539257] LustreError: Skipped 8 previous similar messages
[ 3265.545486] Lustre: lustre-OST0000-osc-ffff9e6dd2328800: Connection to lustre-OST0000 (at 192.168.201.153@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3265.555906] Lustre: Skipped 7 previous similar messages
[ 3289.146272] Lustre: lustre-OST0000-osc-ffff9e6dd2328800: Connection restored to 192.168.201.153@tcp (at 192.168.201.153@tcp)
[ 3289.158524] Lustre: Skipped 15 previous similar messages
[ 3308.129050] Lustre: DEBUG MARKER: oleg153-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3312.079201] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3643.403821] Lustre: DEBUG MARKER: oleg153-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3646.063147] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3891.761573] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 00:46:16 (1783140376)
[ 3909.599801] Lustre: 50366:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783140379/real 1783140379]  req@ffff9e6dc6698380 x1869754216967808/t0(0) o101->lustre-MDT0000-mdc-ffff9e6dd2328800@192.168.201.153@tcp:12/10 lens 576/1152 e 0 to 1 dl 1783140395 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3909.639983] Lustre: 50366:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
[ 3909.649213] Lustre: lustre-MDT0000-mdc-ffff9e6dd2328800: Connection to lustre-MDT0000 (at 192.168.201.153@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3909.661759] Lustre: Skipped 1 previous similar message
[ 3909.734050] Lustre: lustre-MDT0000-mdc-ffff9e6dd2328800: Connection restored to 192.168.201.153@tcp (at 192.168.201.153@tcp)
[ 3909.746258] Lustre: Skipped 1 previous similar message
[ 3917.311854] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 00:46:42 (1783140402)
[ 3943.618973] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 00:47:08 (1783140428)
[ 3969.523240] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 00:47:34 (1783140454)
[ 3969.919707] Lustre: Mounted lustre-client
[ 4000.735137] Lustre: 2350:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783140471/real 1783140471]  req@ffff9e6dc7cdaa00 x1869754216984704/t0(0) o400->lustre-MDT0000-mdc-ffff9e6dd2328800@192.168.201.153@tcp:12/10 lens 224/224 e 0 to 1 dl 1783140487 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4000.773810] LustreError: MGC192.168.201.153@tcp: Connection to MGS (at 192.168.201.153@tcp) was lost; in progress operations using this service will fail
[ 4000.784674] Lustre: 2350:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 4000.814891] LustreError: Skipped 3 previous similar messages
[ 4000.854669] Lustre: Evicted from MGS (at 192.168.201.153@tcp) after server handle changed from 0x604be086e50712c4 to 0x604be086e51b4ae2
[ 4000.862124] Lustre: Skipped 3 previous similar messages
[ 4012.457690] Lustre: DEBUG MARKER: oleg153-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4014.147172] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4016.010365] Lustre: Unmounted lustre-client
[ 4021.013848] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 00:48:26 (1783140506)
[ 4157.407460] Lustre: 2350:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783140627/real 1783140627]  req@ffff9e6dc7bced80 x1869754217016192/t0(0) o4->lustre-OST0000-osc-ffff9e6dd2328800@192.168.201.153@tcp:6/4 lens 488/448 e 0 to 1 dl 1783140643 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4157.443160] Lustre: 2350:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 43 previous similar messages
[ 4311.478876] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 00:53:16 (1783140796)
[ 4358.932455] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 00:54:04 (1783140844)
[ 4361.642897] Lustre: Unmounted lustre-client
[ 4376.991467] Lustre: Mounted lustre-client
[ 4381.476776] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 00:54:26 (1783140866)
[ 4381.704548] LustreError: 56284:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4382.751165] LustreError: 56284:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4382.878663] Lustre: *** cfs_fail_loc=305, val=0***
[ 4403.727231] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 00:54:49 (1783140889)
[ 4404.105480] Lustre: Mounted lustre-client
[ 4405.192910] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4415.480682] Lustre: Unmounted lustre-client
[ 4419.873257] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 00:55:05 (1783140905)
[ 4495.382260] LustreError: lustre-MDT0000-mdc-ffff9e6dc7a68000: operation mds_reint to node 192.168.201.153@tcp failed: rc = -19
[ 4495.392693] LustreError: Skipped 1 previous similar message
[ 4511.713197] Lustre: 2347:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783140982/real 1783140982]  req@ffff9e6dd0ccf100 x1869754219687552/t0(0) o400->MGC192.168.201.153@tcp@192.168.201.153@tcp:26/25 lens 224/224 e 0 to 1 dl 1783140998 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4511.715914] LustreError: MGC192.168.201.153@tcp: Connection to MGS (at 192.168.201.153@tcp) was lost; in progress operations using this service will fail
[ 4511.721930] Lustre: 2347:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 61 previous similar messages
[ 4511.731523] Lustre: Evicted from MGS (at 192.168.201.153@tcp) after server handle changed from 0x604be086e51b6506 to 0x604be086e51e1d40
[ 4511.749745] Lustre: MGC192.168.201.153@tcp: Connection restored to 192.168.201.153@tcp (at 192.168.201.153@tcp)
[ 4511.755187] Lustre: Skipped 24 previous similar messages
[ 4617.036246] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 00:58:22 (1783141102)
[ 4620.790777] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4629.480173] Lustre: lustre-MDT0000-mdc-ffff9e6dc7a68000: Connection to lustre-MDT0000 (at 192.168.201.153@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4629.487148] Lustre: Skipped 24 previous similar messages
[ 4629.490649] LustreError: MGC192.168.201.153@tcp: Connection to MGS (at 192.168.201.153@tcp) was lost; in progress operations using this service will fail
[ 4629.507716] Lustre: Evicted from MGS (at 192.168.201.153@tcp) after server handle changed from 0x604be086e51e1d40 to 0x604be086e522e1d2
[ 4629.529546] LustreError: lustre-MDT0000-mdc-ffff9e6dc7a68000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4638.753128] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 00:58:44 (1783141124)
[ 4638.991017] Lustre: Mounted lustre-client
[ 4644.326758] LustreError: lustre-OST0000-osc-ffff9e6dc7a68000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4644.333970] LustreError: 59967:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff9e6dc7a68000: namespace resource [0x240000400:0x3cc3:0x0].0x0 (ffff9e6dd0d7e600) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4656.199425] Lustre: Unmounted lustre-client
[ 4659.968308] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 00:59:05 (1783141145)
[ 4665.588797] LustreError: lustre-MDT0000-mdc-ffff9e6dc7a68000: operation ldlm_enqueue to node 192.168.201.153@tcp failed: rc = -107
[ 4665.603668] LustreError: lustre-MDT0000-mdc-ffff9e6dc7a68000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4665.620232] LustreError: 60637:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4665.628962] LustreError: 60720:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9e6dc7a68000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff9e6dc8000500) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4671.445576] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 00:59:16 (1783141156)
[ 4671.547045] LustreError: 61335:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4676.576933] LustreError: 61335:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=0
[ 4676.583291] LustreError: 61335:0:(import.c:713:ptlrpc_connect_import_locked()) already connected
[ 4677.584509] LustreError: lustre-MDT0000-mdc-ffff9e6dc7a68000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4677.591545] LustreError: 61359:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4677.625451] LustreError: 61364:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4678.682171] LustreError: 61371:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4678.692440] LustreError: 61371:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4680.829658] LustreError: 61393:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4680.832981] LustreError: 61393:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4685.107700] LustreError: 61437:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4685.111917] LustreError: 61437:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4693.499557] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 00:59:38 (1783141178)
[ 4722.601966] Lustre: DEBUG MARKER: oleg153-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4723.706956] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4730.755875] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 01:00:16 (1783141216)
[ 4757.681410] Lustre: DEBUG MARKER: oleg153-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4758.740326] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4765.870584] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 01:00:51 (1783141251)
[ 4817.398374] Lustre: DEBUG MARKER: oleg153-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4818.280156] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4823.522212] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 01:01:49 (1783141309)
[ 4849.325727] Lustre: DEBUG MARKER: oleg153-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4850.219791] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4853.682054] Lustre: Unmounted lustre-client
[ 4888.221733] Lustre: DEBUG MARKER: oleg153-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4890.352235] Lustre: Mounted lustre-client
[ 4894.282173] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 01:02:59 (1783141379)
[ 4895.649907] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 4896.810383] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 01:03:02 (1783141382)
[ 4914.591718] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 01:03:19 (1783141399)
[ 4915.598589] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 4916.716446] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 01:03:22 (1783141402)
[ 4917.563653] Lustre: *** cfs_fail_loc=805, val=0***
[ 4917.590689] Lustre: Mounted lustre-client
[ 4920.795411] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4938.220183] LustreError: MGC192.168.201.153@tcp: Connection to MGS (at 192.168.201.153@tcp) was lost; in progress operations using this service will fail
[ 4938.237494] Lustre: Evicted from MGS (at 192.168.201.153@tcp) after server handle changed from 0x604be086e522ec4b to 0x604be086e522efd9
[ 4941.100980] Lustre: Unmounted lustre-client
[ 4945.054950] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 01:03:50 (1783141430)
[ 4971.009207] LustreError: 2346:0:(client.c:3447:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff9e6dd07cb100 x1869754221729664/t90194313220(90194313220) o101->lustre-MDT0000-mdc-ffff9e6dc67de800@192.168.201.153@tcp:12/10 lens 664/608 e 0 to 0 dl 1783141473 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 4974.757295] Lustre: DEBUG MARKER: oleg153-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4975.539644] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4984.379432] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 01:04:29 (1783141469)
[ 4986.354848] LustreError: lustre-OST0000-osc-ffff9e6dc67de800: operation ost_write to node 192.168.201.153@tcp failed: rc = -107
[ 4986.371231] LustreError: Skipped 1 previous similar message
[ 4986.397897] LustreError: lustre-OST0000-osc-ffff9e6dc67de800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4986.428597] Lustre: 2349:0:(llite_lib.c:4202:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.153@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 4986.483346] LustreError: 72460:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff9e6dc67de800: namespace resource [0x240000400:0x3d22:0x0].0x0 (ffff9e6dd13c2d00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5002.051107] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 01:04:44 (1783141484)
[ 5005.479844] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5009.308639] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 01:04:53 (1783141493)
[ 5013.402969] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5016.815323] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 01:05:01 (1783141501)
[ 5019.029354] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5022.800369] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 01:05:06 (1783141506)
[ 5026.633934] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5031.764882] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 01:05:13 (1783141513)
[ 5036.785151] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5041.216735] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 01:05:24 (1783141524)
[ 5044.682115] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5048.351922] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 01:05:31 (1783141531)
[ 5051.024431] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5054.120548] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 01:05:38 (1783141538)
[ 5056.177689] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5058.893458] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 01:05:42 (1783141542)
[ 5061.310602] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5063.928973] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 01:05:48 (1783141548)
[ 5066.564054] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5068.505969] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 01:05:53 (1783141553)
[ 5070.364072] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5072.403361] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 01:05:56 (1783141556)
[ 5075.265415] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5077.397336] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 01:06:01 (1783141561)
[ 5110.566298] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 01:06:35 (1783141595)
[ 5140.848928] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 01:07:05 (1783141625)
[ 5141.333494] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5152.221941] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 01:07:16 (1783141636)
[ 5152.597986] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5162.722462] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 01:07:27 (1783141647)
[ 5162.994894] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5171.029283] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 01:07:36 (1783141656)
[ 5171.469483] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5179.559976] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 01:07:44 (1783141664)
[ 5179.847177] Lustre: *** cfs_fail_loc=510, val=3***
[ 5187.462701] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 01:07:52 (1783141672)
[ 5187.718348] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5198.004131] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 01:08:03 (1783141683)
[ 5198.652202] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5206.815420] Lustre: 2350:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783141677/real 1783141677]  req@ffff9e6dc7afb480 x1869754221773952/t0(0) o400->lustre-OST0001-osc-ffff9e6dc67de800@192.168.201.153@tcp:28/4 lens 224/224 e 0 to 1 dl 1783141693 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5206.834225] Lustre: 2350:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 14 previous similar messages
[ 5258.719244] Lustre: lustre-OST0000-osc-ffff9e6dc67de800: Connection to lustre-OST0000 (at 192.168.201.153@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5258.727495] Lustre: Skipped 16 previous similar messages
[ 5264.702641] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 01:09:09 (1783141749)
[ 5265.075849] LustreError: 82660:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5267.924902] LustreError: lustre-MDT0000-mdc-ffff9e6dc67de800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5267.945966] LustreError: 82674:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9e6dc67de800: inode [0x20000a042:0xe:0x0] mdc close failed: rc = -108
[ 5267.993417] LustreError: 82674:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9e6dc67de800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9e6dc8000700) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5268.036155] Lustre: lustre-MDT0000-mdc-ffff9e6dc67de800: Connection restored to 192.168.201.153@tcp (at 192.168.201.153@tcp)
[ 5268.045768] Lustre: Skipped 19 previous similar messages
[ 5269.167210] LustreError: 82660:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5271.285205] LustreError: 82690:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5274.244885] LustreError: lustre-MDT0000-mdc-ffff9e6dc67de800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5274.259814] LustreError: 82704:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9e6dc67de800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff9e6dc8000800) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5274.274803] LustreError: 82704:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5275.367156] LustreError: 82690:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5282.356911] LustreError: lustre-MDT0000-mdc-ffff9e6dc67de800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5282.802486] LustreError: 82746:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5285.943910] LustreError: lustre-MDT0000-mdc-ffff9e6dc67de800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5285.990510] LustreError: 82765:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5285.999863] LustreError: 82765:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 20 previous similar messages
[ 5286.888591] LustreError: 82746:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5286.909294] LustreError: 82746:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9e6dc67de800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5293.033678] LustreError: 82821:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5295.969520] LustreError: lustre-MDT0000-mdc-ffff9e6dc67de800: operation ldlm_enqueue to node 192.168.201.153@tcp failed: rc = -107
[ 5295.988279] LustreError: Skipped 4 previous similar messages
[ 5296.009490] LustreError: lustre-MDT0000-mdc-ffff9e6dc67de800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5296.035516] LustreError: 82835:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9e6dc67de800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9e6dd13c2100) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5296.078603] LustreError: 82835:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 2 previous similar messages
[ 5297.119299] LustreError: 82821:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5304.258124] LustreError: lustre-MDT0000-mdc-ffff9e6dc67de800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5304.695808] LustreError: 82877:0:(ldlm_flock.c:858:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5307.736557] LustreError: lustre-MDT0000-mdc-ffff9e6dc67de800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5308.775143] LustreError: 82877:0:(ldlm_flock.c:858:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5311.985058] LustreError: lustre-MDT0000-mdc-ffff9e6dc67de800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5312.138211] LustreError: 82925:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5312.160250] LustreError: 82925:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5312.968642] LustreError: 82905:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9e6dc67de800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5321.580089] LustreError: 82977:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5321.606058] LustreError: 82977:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5325.525258] LustreError: lustre-MDT0000-mdc-ffff9e6dc67de800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5325.584069] LustreError: 82977:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5325.599868] LustreError: 82977:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5325.636954] LustreError: 82977:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9e6dc67de800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5337.361085] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 01:10:22 (1783141822)
[ 5356.533358] LustreError: 67949:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5356.536428] LustreError: 67949:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5360.599243] LustreError: 67949:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5360.611126] LustreError: 67949:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5366.891137] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 01:10:51 (1783141851)
[ 5416.968359] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 01:11:42 (1783141902)
[ 5479.552874] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 01:12:44 (1783141964)
[ 5511.936614] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 01:13:17 (1783141997)
[ 5512.579520] Lustre: Mounted lustre-client
[ 5635.644743] Lustre: Unmounted lustre-client
[ 5641.002395] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 01:15:26 (1783142126)
[ 5641.523408] LustreError: 87303:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5643.852885] LustreError: lustre-OST0000-osc-ffff9e6dc67de800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5643.876626] LustreError: 87390:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff9e6dc67de800: namespace resource [0x240000400:0x3d4d:0x0].0x0 (ffff9e6dd0219f00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5643.892027] LustreError: 87390:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 3 previous similar messages
[ 5644.143238] LustreError: 87303:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5644.147805] Lustre: 2347:0:(llite_lib.c:4202:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.153@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5649.694822] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 01:15:34 (1783142134)
[ 5711.412782] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 01:16:36 (1783142196)
[ 5712.461795] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 5713.540992] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 01:16:38 (1783142198)
[ 5774.925155] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 5776.166245] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 01:17:41 (1783142261)
[ 5832.672057] Lustre: 89701:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783142263/real 1783142263]  req@ffff9e6dd83e4700 x1869754221919360/t0(0) o36->lustre-MDT0000-mdc-ffff9e6dc67de800@192.168.201.153@tcp:12/10 lens 496/440 e 0 to 1 dl 1783142318 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ln.0' uid:0 gid:0 projid:4294967295
[ 5832.692851] Lustre: 89701:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
[ 5833.666119] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 01:18:39 (1783142319)
[ 5835.104937] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 5836.205361] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 01:18:41 (1783142321)
[ 5837.130427] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 5838.498266] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 01:18:43 (1783142323)
[ 5859.078546] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 01:19:04 (1783142344)
[ 5867.888932] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5888.991298] LustreError: MGC192.168.201.153@tcp: Connection to MGS (at 192.168.201.153@tcp) was lost; in progress operations using this service will fail
[ 5889.007217] LustreError: Skipped 2 previous similar messages
[ 5889.026966] Lustre: lustre-MDT0000-mdc-ffff9e6dc67de800: Connection to lustre-MDT0000 (at 192.168.201.153@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5889.030379] Lustre: Evicted from MGS (at 192.168.201.153@tcp) after server handle changed from 0x604be086e522fa9f to 0x604be086e52313ff
[ 5889.037677] Lustre: Skipped 16 previous similar messages
[ 5889.047872] Lustre: Skipped 2 previous similar messages
[ 5889.053561] Lustre: MGC192.168.201.153@tcp: Connection restored to 192.168.201.153@tcp (at 192.168.201.153@tcp)
[ 5889.059143] Lustre: Skipped 15 previous similar messages
[ 5897.846307] Lustre: DEBUG MARKER: oleg153-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5898.866183] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5905.801463] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 01:19:51 (1783142391)
[ 5907.502413] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 5908.524960] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 01:19:53 (1783142393)
[ 5925.275815] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 01:20:10 (1783142410)
[ 5944.299121] LustreError: MGC192.168.201.153@tcp: Connection to MGS (at 192.168.201.153@tcp) was lost; in progress operations using this service will fail
[ 5944.308444] LustreError: Skipped 1 previous similar message
[ 5944.322172] Lustre: Evicted from MGS (at 192.168.201.153@tcp) after server handle changed from 0x604be086e52317b7 to 0x604be086e5231b3e
[ 5944.330093] Lustre: Skipped 1 previous similar message
[ 5952.103449] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 01:20:37 (1783142437)
[ 5957.029829] LustreError: lustre-OST0000-osc-ffff9e6dc67de800: operation ost_setattr to node 192.168.201.153@tcp failed: rc = -107
[ 5957.040135] LustreError: Skipped 5 previous similar messages
[ 5995.427638] Lustre: DEBUG MARKER: oleg153-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5997.365950] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6077.413450] LustreError: MGC192.168.201.153@tcp: Connection to MGS (at 192.168.201.153@tcp) was lost; in progress operations using this service will fail
[ 6077.429547] Lustre: Evicted from MGS (at 192.168.201.153@tcp) after server handle changed from 0x604be086e5231b3e to 0x604be086e5232929
[ 6085.689401] Lustre: DEBUG MARKER: oleg153-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6086.658813] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6112.566475] Lustre: DEBUG MARKER: oleg153-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6113.993259] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6205.023535] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 01:24:50 (1783142690)
[ 6245.569834] Lustre: DEBUG MARKER: oleg153-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 6247.464903] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6446.324588] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 01:28:51 (1783142931)
[ 6574.056302] LustreError: MGC192.168.201.153@tcp: Connection to MGS (at 192.168.201.153@tcp) was lost; in progress operations using this service will fail
[ 6574.060815] LustreError: Skipped 1 previous similar message
[ 6574.064115] Lustre: lustre-MDT0000-mdc-ffff9e6dc67de800: Connection to lustre-MDT0000 (at 192.168.201.153@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6574.072593] Lustre: Skipped 6 previous similar messages
[ 6574.072879] Lustre: Evicted from MGS (at 192.168.201.153@tcp) after server handle changed from 0x604be086e5232db3 to 0x604be086e53298d3
[ 6574.080131] Lustre: Skipped 1 previous similar message
[ 6574.085054] Lustre: MGC192.168.201.153@tcp: Connection restored to 192.168.201.153@tcp (at 192.168.201.153@tcp)
[ 6574.089106] Lustre: Skipped 9 previous similar messages
[ 6597.746986] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 01:31:23 (1783143083)
[ 6598.683194] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 6599.736553] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 01:31:25 (1783143085)
[ 6601.009419] LustreError: lustre-MDT0000-mdc-ffff9e6dc67de800: operation ldlm_enqueue to node 192.168.201.153@tcp failed: rc = -107
[ 6601.020672] LustreError: Skipped 179 previous similar messages
[ 6601.033490] LustreError: lustre-MDT0000-mdc-ffff9e6dc67de800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6605.984721] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 01:31:31 (1783143091)
[ 6772.522875] LustreError: lustre-OST0000-osc-ffff9e6dc67de800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6773.747372] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 01:34:19 (1783143259)
[ 6797.215199] Lustre: 2350:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783143263/real 1783143263]  req@ffff9e6dd0a17100 x1869754239346944/t0(0) o4->lustre-OST0000-osc-ffff9e6dc67de800@192.168.201.153@tcp:6/4 lens 4584/448 e 0 to 1 dl 1783143283 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 6797.228308] Lustre: 2350:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
[ 6811.214819] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 01:34:56 (1783143296)
[ 6811.969216] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 6812.874751] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 01:34:58 (1783143298)
[ 6813.780629] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 6814.834910] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 01:35:00 (1783143300)
[ 6842.214669] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 01:35:27 (1783143327)
[ 6874.082375] LustreError: MGC192.168.201.153@tcp: Connection to MGS (at 192.168.201.153@tcp) was lost; in progress operations using this service will fail
[ 6874.097969] Lustre: Evicted from MGS (at 192.168.201.153@tcp) after server handle changed from 0x604be086e53298d3 to 0x604be086e532c13d
[ 6874.103394] Lustre: Skipped 1 previous similar message
[ 6880.175214] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 01:36:05 (1783143365)
[ 6880.909700] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 6881.636707] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 01:36:07 (1783143367)
[ 6882.269947] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 6883.106709] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 01:36:08 (1783143368)
[ 6888.887160] Lustre: Unmounted lustre-client
[ 6889.600688] Lustre: Mounted lustre-client
[ 6891.079389] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6910.343917] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2