[ 2002.785302] LustreError: lustre-OST0000-osc-ffff8aa3071d6000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2002.810847] Lustre: 2354:0:(llite_lib.c:4327:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.113@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 2011.775304] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 09:20:58 (1783948858)
[ 2013.796496] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 2016.328224] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 09:21:02 (1783948862)
[ 2018.944302] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 2021.324857] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 09:21:07 (1783948867)
[ 2026.270652] LustreError: lustre-MDT0000-mdc-ffff8aa3071d6000: operation mds_reint to node 192.168.201.113@tcp failed: rc = -19
[ 2026.288709] LustreError: Skipped 1 previous similar message
[ 2044.401095] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 192.168.201.113@tcp) was lost; in progress operations using this service will fail
[ 2054.659219] Lustre: Evicted from MGS (at 192.168.201.113@tcp) after server handle changed from 0x3ca3296215458674 to 0x3ca3296215459f72
[ 2151.298680] LustreError: lustre-MDT0000-mdc-ffff8aa3071d6000: operation ldlm_enqueue to node 192.168.201.113@tcp failed: rc = -19
[ 2151.307604] LustreError: Skipped 1 previous similar message
[ 2170.344690] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 192.168.201.113@tcp) was lost; in progress operations using this service will fail
[ 2170.376305] Lustre: Evicted from MGS (at 192.168.201.113@tcp) after server handle changed from 0x3ca3296215459f72 to 0x3ca329621547427d
[ 2183.565730] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 09:23:50 (1783949030)
[ 2184.137807] Lustre: *** cfs_fail_loc=305, val=0***
[ 2184.141373] Lustre: Skipped 4 previous similar messages
[ 2226.679947] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 192.168.201.113@tcp) was lost; in progress operations using this service will fail
[ 2226.712216] Lustre: Evicted from MGS (at 192.168.201.113@tcp) after server handle changed from 0x3ca329621547427d to 0x3ca32962154769c1
[ 2238.793866] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2240.453704] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2249.266596] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 09:24:56 (1783949096)
[ 2262.506884] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 192.168.201.113@tcp) was lost; in progress operations using this service will fail
[ 2262.527235] Lustre: Evicted from MGS (at 192.168.201.113@tcp) after server handle changed from 0x3ca32962154769c1 to 0x3ca3296215476f63
[ 2271.900501] LustreError: lustre-MDT0000-mdc-ffff8aa3071d6000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2288.329287] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 09:25:35 (1783949135)
[ 2307.568460] LustreError: lustre-OST0000-osc-ffff8aa3071d6000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2322.952095] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 09:26:10 (1783949170)
[ 2336.825106] LustreError: lustre-MDT0000-mdc-ffff8aa3071d6000: operation mds_close to node 192.168.201.113@tcp failed: rc = -19
[ 2336.835443] LustreError: Skipped 2 previous similar messages
[ 2354.655721] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 192.168.201.113@tcp) was lost; in progress operations using this service will fail
[ 2363.829755] Lustre: Evicted from MGS (at 192.168.201.113@tcp) after server handle changed from 0x3ca3296215476f63 to 0x3ca329621547bbfa
[ 2381.997818] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2384.613254] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2470.393565] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 192.168.201.113@tcp) was lost; in progress operations using this service will fail
[ 2470.471474] Lustre: Evicted from MGS (at 192.168.201.113@tcp) after server handle changed from 0x3ca329621547bbfa to 0x3ca3296215494677
[ 2488.871595] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2491.872535] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2559.216628] LustreError: lustre-MDT0000-mdc-ffff8aa3071d6000: operation mds_close to node 192.168.201.113@tcp failed: rc = -19
[ 2559.226763] Lustre: lustre-MDT0000-mdc-ffff8aa3071d6000: Connection to lustre-MDT0000 (at 192.168.201.113@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2559.230000] LustreError: Skipped 2 previous similar messages
[ 2559.269850] Lustre: Skipped 9 previous similar messages
[ 2577.951785] Lustre: 2357:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783949410/real 1783949410]  req@ffff8aa306639880 x1870603968390272/t0(0) o400->MGC192.168.201.113@tcp@192.168.201.113@tcp:26/25 lens 224/224 e 0 to 1 dl 1783949426 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2578.009724] Lustre: 2357:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 9 previous similar messages
[ 2578.029685] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 192.168.201.113@tcp) was lost; in progress operations using this service will fail
[ 2588.294160] Lustre: Evicted from MGS (at 192.168.201.113@tcp) after server handle changed from 0x3ca3296215494677 to 0x3ca32962154aff23
[ 2588.327919] Lustre: MGC192.168.201.113@tcp: Connection restored to 192.168.201.113@tcp (at 192.168.201.113@tcp)
[ 2588.338803] Lustre: Skipped 15 previous similar messages
[ 2589.757625] Lustre: 15925:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.113@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2612.945406] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2615.325839] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2645.498346] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 09:31:32 (1783949492)
[ 2669.557158] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 192.168.201.113@tcp) was lost; in progress operations using this service will fail
[ 2669.595909] Lustre: Evicted from MGS (at 192.168.201.113@tcp) after server handle changed from 0x3ca32962154aff23 to 0x3ca32962154bd45d
[ 2681.874378] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2717.741327] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2757.911958] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2794.799262] Lustre: 15925:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.113@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2808.481834] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2831.756872] LustreError: lustre-MDT0000-mdc-ffff8aa3071d6000: operation ldlm_enqueue to node 192.168.201.113@tcp failed: rc = -19
[ 2831.778297] LustreError: Skipped 7 previous similar messages
[ 2848.735689] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 192.168.201.113@tcp) was lost; in progress operations using this service will fail
[ 2848.746093] LustreError: Skipped 3 previous similar messages
[ 2859.041707] Lustre: Evicted from MGS (at 192.168.201.113@tcp) after server handle changed from 0x3ca32962154c8a5d to 0x3ca32962154cfa5d
[ 2859.065855] Lustre: Skipped 3 previous similar messages
[ 2861.826660] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2921.542415] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 3013.783846] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 09:37:40 (1783949860)
[ 3071.268388] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3073.381821] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3359.179853] LustreError: lustre-OST0000-osc-ffff8aa3071d6000: operation ldlm_enqueue to node 192.168.201.113@tcp failed: rc = -107
[ 3359.197588] LustreError: Skipped 3 previous similar messages
[ 3359.206445] Lustre: lustre-OST0000-osc-ffff8aa3071d6000: Connection to lustre-OST0000 (at 192.168.201.113@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3359.229605] Lustre: Skipped 8 previous similar messages
[ 3383.888349] Lustre: lustre-OST0000-osc-ffff8aa3071d6000: Connection restored to 192.168.201.113@tcp (at 192.168.201.113@tcp)
[ 3383.902591] Lustre: Skipped 16 previous similar messages
[ 3403.813847] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3406.316701] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3735.791739] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3737.824555] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3988.170773] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 09:53:54 (1783950834)
[ 4005.855413] Lustre: 50393:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783950838/real 1783950838]  req@ffff8aa306768a80 x1870603987704448/t0(0) o101->lustre-MDT0000-mdc-ffff8aa3071d6000@192.168.201.113@tcp:12/10 lens 576/1152 e 0 to 1 dl 1783950854 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 4005.889793] Lustre: 50393:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
[ 4005.909321] Lustre: lustre-MDT0000-mdc-ffff8aa3071d6000: Connection to lustre-MDT0000 (at 192.168.201.113@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4005.934860] Lustre: Skipped 1 previous similar message
[ 4005.993612] Lustre: lustre-MDT0000-mdc-ffff8aa3071d6000: Connection restored to 192.168.201.113@tcp (at 192.168.201.113@tcp)
[ 4006.016076] Lustre: Skipped 1 previous similar message
[ 4015.189730] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 09:54:21 (1783950861)
[ 4043.863367] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 09:54:50 (1783950890)
[ 4071.767263] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 09:55:18 (1783950918)
[ 4072.172379] Lustre: Mounted lustre-client
[ 4104.159476] Lustre: 2356:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783950936/real 1783950936]  req@ffff8aa305b56680 x1870603987722496/t0(0) o400->lustre-MDT0000-mdc-ffff8aa3071d6000@192.168.201.113@tcp:12/10 lens 224/224 e 0 to 1 dl 1783950952 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4104.189593] Lustre: 2356:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 4104.208245] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 192.168.201.113@tcp) was lost; in progress operations using this service will fail
[ 4104.220955] LustreError: Skipped 2 previous similar messages
[ 4114.438852] Lustre: Evicted from MGS (at 192.168.201.113@tcp) after server handle changed from 0x3ca32962154e0203 to 0x3ca3296215618325
[ 4114.451387] Lustre: Skipped 2 previous similar messages
[ 4135.950330] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4137.694512] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4139.643190] Lustre: Unmounted lustre-client
[ 4147.072333] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 09:56:33 (1783950993)
[ 4268.832234] Lustre: 2356:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783951101/real 1783951101]  req@ffff8aa3065aaa00 x1870603987756672/t0(0) o4->lustre-OST0000-osc-ffff8aa3071d6000@192.168.201.113@tcp:6/4 lens 488/448 e 0 to 1 dl 1783951117 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4268.877425] Lustre: 2356:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 53 previous similar messages
[ 4444.988579] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 10:01:31 (1783951291)
[ 4498.198413] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 10:02:24 (1783951344)
[ 4502.838729] Lustre: Unmounted lustre-client
[ 4525.201031] Lustre: Mounted lustre-client
[ 4532.583393] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 10:02:58 (1783951378)
[ 4532.911770] LustreError: 56320:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4533.968104] LustreError: 56320:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4534.185758] Lustre: *** cfs_fail_loc=305, val=0***
[ 4558.953935] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 10:03:25 (1783951405)
[ 4559.375089] Lustre: Mounted lustre-client
[ 4561.268335] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4571.600933] Lustre: Unmounted lustre-client
[ 4580.738488] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 10:03:46 (1783951426)
[ 4685.944933] LustreError: lustre-MDT0000-mdc-ffff8aa306747800: operation mds_reint to node 192.168.201.113@tcp failed: rc = -19
[ 4685.960027] LustreError: Skipped 1 previous similar message
[ 4685.966015] Lustre: lustre-MDT0000-mdc-ffff8aa306747800: Connection to lustre-MDT0000 (at 192.168.201.113@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4685.990527] Lustre: Skipped 23 previous similar messages
[ 4703.711215] Lustre: 2356:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783951535/real 1783951535]  req@ffff8aa305e67480 x1870603990425856/t0(0) o400->MGC192.168.201.113@tcp@192.168.201.113@tcp:26/25 lens 224/224 e 0 to 1 dl 1783951551 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4703.734652] Lustre: 2356:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 89 previous similar messages
[ 4703.743114] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 192.168.201.113@tcp) was lost; in progress operations using this service will fail
[ 4713.972896] Lustre: Evicted from MGS (at 192.168.201.113@tcp) after server handle changed from 0x3ca3296215619d9d to 0x3ca32962156453a7
[ 4713.991750] Lustre: MGC192.168.201.113@tcp: Connection restored to 192.168.201.113@tcp (at 192.168.201.113@tcp)
[ 4714.014239] Lustre: Skipped 24 previous similar messages
[ 4879.837206] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 10:08:46 (1783951726)
[ 4885.916532] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4901.368873] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 192.168.201.113@tcp) was lost; in progress operations using this service will fail
[ 4901.408962] Lustre: Evicted from MGS (at 192.168.201.113@tcp) after server handle changed from 0x3ca32962156453a7 to 0x3ca3296215691cc3
[ 4901.441405] LustreError: lustre-MDT0000-mdc-ffff8aa306747800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4914.386603] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 10:09:21 (1783951761)
[ 4915.112569] Lustre: Mounted lustre-client
[ 4922.601697] LustreError: lustre-OST0000-osc-ffff8aa306747800: operation ldlm_enqueue to node 192.168.201.113@tcp failed: rc = -107
[ 4922.629476] LustreError: lustre-OST0000-osc-ffff8aa306747800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4933.931776] Lustre: Unmounted lustre-client
[ 4941.020663] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 10:09:48 (1783951788)
[ 4948.071673] LustreError: lustre-MDT0000-mdc-ffff8aa306747800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4948.087567] LustreError: 60784:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8aa306747800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff8aa306d32b00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4948.100834] LustreError: 60700:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4954.907942] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 10:10:02 (1783951802)
[ 4955.129657] LustreError: 61404:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4959.121406] LustreError: lustre-MDT0000-mdc-ffff8aa306747800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4959.144472] LustreError: 61421:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4959.161719] LustreError: 61404:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=973
[ 4959.172372] LustreError: 61404:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting
[ 4959.222560] LustreError: 61426:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4960.324861] LustreError: 61432:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4960.339787] LustreError: 61432:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4962.671478] LustreError: 61455:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4962.679952] LustreError: 61455:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4967.273166] LustreError: 61500:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4967.287576] LustreError: 61500:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4977.172275] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 10:10:24 (1783951824)
[ 5016.528942] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5018.494507] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5028.333374] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 10:11:15 (1783951875)
[ 5070.741776] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5072.417679] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5084.890918] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 10:12:11 (1783951931)
[ 5149.296860] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5151.073548] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5159.673536] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 10:13:26 (1783952006)
[ 5198.963916] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5200.480366] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5207.452920] Lustre: Unmounted lustre-client
[ 5261.645546] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5263.971852] Lustre: Mounted lustre-client
[ 5270.561543] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 10:15:17 (1783952117)
[ 5273.744169] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5276.044484] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 10:15:22 (1783952122)
[ 5308.842245] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 10:15:55 (1783952155)
[ 5310.225416] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5312.291460] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 10:15:59 (1783952159)
[ 5314.017963] Lustre: *** cfs_fail_loc=805, val=0***
[ 5314.081564] Lustre: Mounted lustre-client
[ 5319.611747] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5339.632575] Lustre: lustre-MDT0000-mdc-ffff8aa3106c1800: Connection to lustre-MDT0000 (at 192.168.201.113@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5339.653489] Lustre: Skipped 11 previous similar messages
[ 5339.658962] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 192.168.201.113@tcp) was lost; in progress operations using this service will fail
[ 5339.694967] Lustre: Evicted from MGS (at 192.168.201.113@tcp) after server handle changed from 0x3ca32962156926d3 to 0x3ca32962156929ea
[ 5339.723969] Lustre: MGC192.168.201.113@tcp: Connection restored to 192.168.201.113@tcp (at 192.168.201.113@tcp)
[ 5339.730730] Lustre: Skipped 11 previous similar messages
[ 5340.575177] Lustre: 2356:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783952172/real 1783952172]  req@ffff8aa320ff4e00 x1870603992475904/t0(0) o400->lustre-MDT0000-mdc-ffff8aa318b39800@192.168.201.113@tcp:12/10 lens 224/224 e 0 to 1 dl 1783952188 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5340.609400] Lustre: 2356:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5350.836655] Lustre: Unmounted lustre-client
[ 5358.480620] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 10:16:45 (1783952205)
[ 5401.141092] LustreError: 2353:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff8aa306db8a80 x1870603992487680/t94489280517(94489280517) o101->lustre-MDT0000-mdc-ffff8aa3106c1800@192.168.201.113@tcp:12/10 lens 648/608 e 0 to 0 dl 1783952265 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5409.295161] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5411.050380] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5420.171344] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 10:17:46 (1783952266)
[ 5421.562134] LustreError: lustre-OST0000-osc-ffff8aa3106c1800: operation ost_write to node 192.168.201.113@tcp failed: rc = -107
[ 5421.566363] LustreError: Skipped 2 previous similar messages
[ 5421.583028] LustreError: lustre-OST0000-osc-ffff8aa3106c1800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5421.595579] Lustre: 2356:0:(llite_lib.c:4327:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.113@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5421.635363] LustreError: 72536:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff8aa3106c1800: namespace resource [0x240000400:0x3b62:0x0].0x0 (ffff8aa306d32800) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5430.734880] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 10:17:57 (1783952277)
[ 5432.293867] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5434.339921] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 10:18:01 (1783952281)
[ 5435.687329] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5437.770644] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 10:18:04 (1783952284)
[ 5439.369921] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5441.257145] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 10:18:07 (1783952287)
[ 5442.959520] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5444.157335] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 10:18:11 (1783952291)
[ 5445.983162] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5448.229772] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 10:18:14 (1783952294)
[ 5450.005868] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5451.444802] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 10:18:18 (1783952298)
[ 5453.106699] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5455.170547] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 10:18:21 (1783952301)
[ 5456.791043] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5458.820334] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 10:18:25 (1783952305)
[ 5460.376311] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5462.438382] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 10:18:29 (1783952309)
[ 5463.759777] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5465.620193] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 10:18:32 (1783952312)
[ 5467.438283] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5469.014846] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 10:18:36 (1783952316)
[ 5471.889960] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5473.544606] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 10:18:40 (1783952320)
[ 5505.722475] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 10:19:12 (1783952352)
[ 5536.816666] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 10:19:43 (1783952383)
[ 5537.514800] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5548.269912] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 10:19:55 (1783952395)
[ 5548.620362] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5559.885955] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 10:20:06 (1783952406)
[ 5560.419553] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5568.328848] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 10:20:15 (1783952415)
[ 5568.809091] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5577.470399] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 10:20:24 (1783952424)
[ 5578.035623] Lustre: *** cfs_fail_loc=510, val=3***
[ 5588.163719] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 10:20:34 (1783952434)
[ 5588.616654] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5599.381849] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 10:20:46 (1783952446)
[ 5600.042738] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5666.071930] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 10:21:53 (1783952513)
[ 5666.625761] LustreError: 82729:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5669.456199] LustreError: lustre-MDT0000-mdc-ffff8aa3106c1800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5669.482553] LustreError: 82744:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8aa3106c1800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8aa306d32d00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5670.711636] LustreError: 82729:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5672.829216] LustreError: 82760:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5676.074093] LustreError: lustre-MDT0000-mdc-ffff8aa3106c1800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5676.107735] LustreError: 82774:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8aa3106c1800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8aa305ea0000) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5676.141943] LustreError: 82774:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5676.839165] LustreError: 82760:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5684.017297] LustreError: lustre-MDT0000-mdc-ffff8aa3106c1800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5684.526425] LustreError: 82816:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5687.624378] LustreError: lustre-MDT0000-mdc-ffff8aa3106c1800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5687.715643] LustreError: 82836:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5687.724438] LustreError: 82836:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5688.591282] LustreError: 82816:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5688.607640] LustreError: 82816:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8aa3106c1800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5688.813395] LustreError: 82842:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5688.826212] LustreError: 82842:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5691.213302] LustreError: 82864:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5691.222165] LustreError: 82864:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5694.982097] LustreError: 82891:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5698.243133] LustreError: lustre-MDT0000-mdc-ffff8aa3106c1800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5698.259929] LustreError: 82906:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8aa3106c1800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff8aa306d32b00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5698.289306] LustreError: 82906:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5699.079136] LustreError: 82891:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5705.999590] LustreError: lustre-MDT0000-mdc-ffff8aa3106c1800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5706.564761] LustreError: 82947:0:(ldlm_flock.c:858:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5709.526830] LustreError: lustre-MDT0000-mdc-ffff8aa3106c1800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5709.558568] LustreError: 82962:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8aa3106c1800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8aa305ea0b00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5709.602222] LustreError: 82962:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 2 previous similar messages
[ 5710.631161] LustreError: 82947:0:(ldlm_flock.c:858:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5713.957593] LustreError: lustre-MDT0000-mdc-ffff8aa3106c1800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5714.047176] LustreError: 82995:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5714.054648] LustreError: 82995:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5714.816097] LustreError: 82975:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8aa3106c1800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5723.201396] LustreError: 83047:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5723.208566] LustreError: 83047:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5727.225602] LustreError: lustre-MDT0000-mdc-ffff8aa3106c1800: operation ldlm_enqueue to node 192.168.201.113@tcp failed: rc = -107
[ 5727.235410] LustreError: Skipped 8 previous similar messages
[ 5727.247271] LustreError: lustre-MDT0000-mdc-ffff8aa3106c1800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5727.303326] LustreError: 83047:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5727.311688] LustreError: 83047:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5727.318916] LustreError: 83047:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8aa3106c1800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5727.322514] LustreError: 83069:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5727.343450] LustreError: 83069:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5738.543275] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 10:23:05 (1783952585)
[ 5757.912754] LustreError: 68016:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5757.927993] LustreError: 68016:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5762.007203] LustreError: 68016:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5762.013617] LustreError: 68016:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5769.840173] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 10:23:36 (1783952616)
[ 5822.108691] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 10:24:28 (1783952668)
[ 5887.493660] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 10:25:34 (1783952734)
[ 5922.468296] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 10:26:09 (1783952769)
[ 5923.217040] Lustre: Mounted lustre-client
[ 6046.875352] Lustre: Unmounted lustre-client
[ 6053.570693] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 10:28:20 (1783952900)
[ 6054.332380] LustreError: 87376:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 6056.880838] Lustre: lustre-OST0000-osc-ffff8aa3106c1800: Connection to lustre-OST0000 (at 192.168.201.113@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6056.891706] Lustre: Skipped 17 previous similar messages
[ 6056.903916] LustreError: lustre-OST0000-osc-ffff8aa3106c1800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6056.911282] LustreError: 87459:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff8aa3106c1800: namespace resource [0x240000400:0x3b8d:0x0].0x0 (ffff8aa306caaa00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6056.918074] LustreError: 87459:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 6057.263149] LustreError: 87376:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout interrupted
[ 6057.270475] Lustre: 2354:0:(llite_lib.c:4327:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.113@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 6063.591779] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 10:28:30 (1783952910)
[ 6121.439719] Lustre: 88043:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783952914/real 1783952914]  req@ffff8aa306db9180 x1870603992662272/t0(0) o101->lustre-MDT0000-mdc-ffff8aa3106c1800@192.168.201.113@tcp:12/10 lens 328/344 e 0 to 1 dl 1783952969 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 6121.485402] Lustre: 88043:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 17 previous similar messages
[ 6121.513428] Lustre: lustre-MDT0000-mdc-ffff8aa3106c1800: Connection restored to 192.168.201.113@tcp (at 192.168.201.113@tcp)
[ 6121.524176] Lustre: Skipped 18 previous similar messages
[ 6129.807765] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 10:29:36 (1783952976)
[ 6131.767192] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 6134.417726] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 10:29:40 (1783952980)
[ 6198.506383] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6200.013770] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 10:30:47 (1783953047)
[ 6259.065626] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 10:31:46 (1783953106)
[ 6261.043521] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6262.903274] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 10:31:49 (1783953109)
[ 6264.370564] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6266.061899] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 10:31:52 (1783953112)
[ 6304.686965] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 10:32:31 (1783953151)
[ 6315.982724] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6340.575726] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 192.168.201.113@tcp) was lost; in progress operations using this service will fail
[ 6340.588656] LustreError: Skipped 2 previous similar messages
[ 6349.807990] Lustre: Evicted from MGS (at 192.168.201.113@tcp) after server handle changed from 0x3ca329621569350b to 0x3ca3296215694d8b
[ 6349.818790] Lustre: Skipped 2 previous similar messages
[ 6364.680829] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6366.463861] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6378.331746] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 10:33:45 (1783953225)
[ 6380.802853] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6382.683319] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 10:33:49 (1783953229)
[ 6396.915030] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 192.168.201.113@tcp) was lost; in progress operations using this service will fail
[ 6396.960792] Lustre: Evicted from MGS (at 192.168.201.113@tcp) after server handle changed from 0x3ca3296215694d8b to 0x3ca3296215695200
[ 6416.292317] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 10:34:22 (1783953262)
[ 6471.377366] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 10:35:18 (1783953318)
[ 6483.153748] LustreError: lustre-OST0000-osc-ffff8aa3106c1800: operation ost_setattr to node 192.168.201.113@tcp failed: rc = -19
[ 6483.169866] LustreError: Skipped 1 previous similar message
[ 6542.296408] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6547.964311] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6637.535524] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 192.168.201.113@tcp) was lost; in progress operations using this service will fail
[ 6637.564643] LustreError: Skipped 1 previous similar message
[ 6646.769265] Lustre: Evicted from MGS (at 192.168.201.113@tcp) after server handle changed from 0x3ca329621569552c to 0x3ca329621569627d
[ 6646.776344] Lustre: Skipped 1 previous similar message
[ 6663.347384] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6665.238865] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6701.648034] Lustre: lustre-MDT0000-mdc-ffff8aa3106c1800: Connection to lustre-MDT0000 (at 192.168.201.113@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6701.678891] Lustre: Skipped 8 previous similar messages
[ 6717.854406] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6719.734660] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6732.258176] Lustre: 2357:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783953520/real 1783953520]  req@ffff8aa305cd5180 x1870603999189376/t0(0) o400->lustre-MDT0000-mdc-ffff8aa3106c1800@192.168.201.113@tcp:12/10 lens 224/224 e 0 to 1 dl 1783953580 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6732.288072] Lustre: 2357:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 25 previous similar messages
[ 6875.250921] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 10:42:01 (1783953721)
[ 6942.665992] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6946.245599] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 7234.772725] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 10:48:01 (1783954081)
[ 7430.120254] Lustre: lustre-MDT0000-mdc-ffff8aa3106c1800: Connection to lustre-MDT0000 (at 192.168.201.113@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7430.142688] Lustre: Skipped 1 previous similar message
[ 7445.487462] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 192.168.201.113@tcp) was lost; in progress operations using this service will fail
[ 7445.497771] LustreError: Skipped 1 previous similar message
[ 7445.505750] Lustre: Evicted from MGS (at 192.168.201.113@tcp) after server handle changed from 0x3ca3296215696689 to 0x3ca3296215790a90
[ 7445.509647] Lustre: Skipped 1 previous similar message
[ 7445.512243] Lustre: MGC192.168.201.113@tcp: Connection restored to 192.168.201.113@tcp (at 192.168.201.113@tcp)
[ 7445.516709] Lustre: Skipped 12 previous similar messages
[ 7486.995836] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 10:52:13 (1783954333)
[ 7488.920266] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7491.689399] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 10:52:17 (1783954337)
[ 7494.235756] LustreError: lustre-MDT0000-mdc-ffff8aa3106c1800: operation ldlm_enqueue to node 192.168.201.113@tcp failed: rc = -107
[ 7494.249612] LustreError: Skipped 35 previous similar messages
[ 7494.264816] LustreError: lustre-MDT0000-mdc-ffff8aa3106c1800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7504.040944] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 10:52:30 (1783954350)
[ 7672.811080] Lustre: Evicted from lustre-OST0000_UUID (at 192.168.201.113@tcp) after server handle changed from 0x3ca3296215790e8e to 0x3ca3296215790f98
[ 7672.821714] LustreError: lustre-OST0000-osc-ffff8aa3106c1800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7676.247796] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 10:55:22 (1783954522)
[ 7703.455190] Lustre: 2356:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783954531/real 1783954531]  req@ffff8aa320049500 x1870604010377472/t0(0) o4->lustre-OST0000-osc-ffff8aa3106c1800@192.168.201.113@tcp:6/4 lens 4584/448 e 0 to 1 dl 1783954551 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7703.505589] Lustre: 2356:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
[ 7703.552252] Lustre: lustre-OST0000-osc-ffff8aa3106c1800: Connection restored to 192.168.201.113@tcp (at 192.168.201.113@tcp)
[ 7703.564587] Lustre: Skipped 2 previous similar messages
[ 7723.068909] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 10:56:09 (1783954569)
[ 7724.765353] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7726.553649] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 10:56:13 (1783954573)
[ 7728.510778] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7730.846590] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 10:56:17 (1783954577)
[ 7767.719218] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 10:56:54 (1783954614)
[ 7811.063357] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 192.168.201.113@tcp) was lost; in progress operations using this service will fail
[ 7811.096596] Lustre: Evicted from MGS (at 192.168.201.113@tcp) after server handle changed from 0x3ca3296215790a90 to 0x3ca32962157932fa
[ 7825.962647] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 10:57:52 (1783954672)
[ 7827.463970] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7828.901327] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 10:57:56 (1783954676)
[ 7830.355298] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7832.613977] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 10:57:59 (1783954679)
[ 7846.575260] Lustre: Unmounted lustre-client
[ 7848.889579] Lustre: Mounted lustre-client
[ 7851.720046] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7870.433552] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 192.168.201.113@tcp) was lost; in progress operations using this service will fail
[ 7874.579590] Lustre: lustre-MDT0000-mdc-ffff8aa3064e4800: Connection restored to 192.168.201.113@tcp (at 192.168.201.113@tcp)
[ 7874.584683] Lustre: Skipped 2 previous similar messages
[ 7880.232240] Lustre: Evicted from MGS (at 192.168.201.113@tcp) after server handle changed from 0x3ca3296215793bf2 to 0x3ca329621579468e
[ 7881.718398] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2