[ 1942.523607] LustreError: lustre-OST0000-osc-ffff9ad046ce7000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1942.550706] Lustre: 2377:0:(llite_lib.c:4344:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.133@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -5)
[ 1951.093363] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 02:32:55 (1788330775)
[ 1953.934214] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1956.121219] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 02:33:00 (1788330780)
[ 1958.071292] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1960.018239] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 02:33:04 (1788330784)
[ 1963.988568] LustreError: lustre-MDT0000-mdc-ffff9ad046ce7000: operation ldlm_enqueue to node 192.168.203.133@tcp failed: rc = -19
[ 1963.998546] LustreError: Skipped 3 previous similar messages
[ 1983.456996] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 192.168.203.133@tcp) was lost; in progress operations using this service will fail
[ 1983.498981] Lustre: Evicted from MGS (at 192.168.203.133@tcp) after server handle changed from 0xfcfbceb1648c7030 to 0xfcfbceb1648c8a5b
[ 1984.557140] Lustre: 15945:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.133@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2093.360344] LustreError: lustre-MDT0000-mdc-ffff9ad046ce7000: operation mds_reint to node 192.168.203.133@tcp failed: rc = -19
[ 2093.365562] LustreError: Skipped 1 previous similar message
[ 2110.431202] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 192.168.203.133@tcp) was lost; in progress operations using this service will fail
[ 2120.645655] Lustre: Evicted from MGS (at 192.168.203.133@tcp) after server handle changed from 0xfcfbceb1648c8a5b to 0xfcfbceb1648e8d4b
[ 2134.715300] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 02:35:59 (1788330959)
[ 2135.272584] Lustre: *** cfs_fail_loc=305, val=0***
[ 2135.278318] Lustre: Skipped 2 previous similar messages
[ 2174.943239] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 192.168.203.133@tcp) was lost; in progress operations using this service will fail
[ 2185.198695] Lustre: Evicted from MGS (at 192.168.203.133@tcp) after server handle changed from 0xfcfbceb1648e8d4b to 0xfcfbceb1648e8fcf
[ 2185.218471] Lustre: MGC192.168.203.133@tcp: Connection restored to 192.168.203.133@tcp (at 192.168.203.133@tcp)
[ 2185.227293] Lustre: Skipped 13 previous similar messages
[ 2187.967402] Lustre: lustre-MDT0000-mdc-ffff9ad046ce7000: Connection to lustre-MDT0000 (at 192.168.203.133@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2187.984911] Lustre: Skipped 11 previous similar messages
[ 2200.083419] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2201.648602] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2209.286883] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 02:37:14 (1788331034)
[ 2226.153539] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 192.168.203.133@tcp) was lost; in progress operations using this service will fail
[ 2226.180255] Lustre: Evicted from MGS (at 192.168.203.133@tcp) after server handle changed from 0xfcfbceb1648e8fcf to 0xfcfbceb1648e95da
[ 2226.299138] LustreError: lustre-MDT0000-mdc-ffff9ad046ce7000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2244.843693] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 02:37:49 (1788331069)
[ 2261.993534] LustreError: lustre-OST0000-osc-ffff9ad046ce7000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2276.906891] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 02:38:21 (1788331101)
[ 2290.144883] LustreError: lustre-MDT0000-mdc-ffff9ad046ce7000: operation mds_reint to node 192.168.203.133@tcp failed: rc = -19
[ 2308.063228] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 192.168.203.133@tcp) was lost; in progress operations using this service will fail
[ 2318.252349] Lustre: Evicted from MGS (at 192.168.203.133@tcp) after server handle changed from 0xfcfbceb1648e95da to 0xfcfbceb1648ef055
[ 2336.217361] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2338.657257] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2423.796402] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 192.168.203.133@tcp) was lost; in progress operations using this service will fail
[ 2423.817572] Lustre: Evicted from MGS (at 192.168.203.133@tcp) after server handle changed from 0xfcfbceb1648ef055 to 0xfcfbceb16490a20f
[ 2425.135805] Lustre: 15945:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.133@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2450.817850] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2452.885594] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2519.059105] LustreError: lustre-MDT0000-mdc-ffff9ad046ce7000: operation mds_reint to node 192.168.203.133@tcp failed: rc = -19
[ 2519.066406] LustreError: Skipped 2 previous similar messages
[ 2537.439496] Lustre: 2378:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788331347/real 1788331347]  req@ffff9ad045adc000 x1875198778409216/t0(0) o400->MGC192.168.203.133@tcp@192.168.203.133@tcp:26/25 lens 224/224 e 0 to 1 dl 1788331363 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2537.482562] Lustre: 2378:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 17 previous similar messages
[ 2537.495631] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 192.168.203.133@tcp) was lost; in progress operations using this service will fail
[ 2547.691434] Lustre: Evicted from MGS (at 192.168.203.133@tcp) after server handle changed from 0xfcfbceb16490a20f to 0xfcfbceb164924f7e
[ 2566.552408] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2568.827611] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2598.830905] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 02:43:43 (1788331423)
[ 2622.434241] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 192.168.203.133@tcp) was lost; in progress operations using this service will fail
[ 2622.458629] Lustre: Evicted from MGS (at 192.168.203.133@tcp) after server handle changed from 0xfcfbceb164924f7e to 0xfcfbceb164932218
[ 2633.791898] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2669.766651] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2714.980836] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2756.075360] Lustre: Evicted from MGS (at 192.168.203.133@tcp) after server handle changed from 0xfcfbceb16493cd75 to 0xfcfbceb1649417b2
[ 2756.089813] Lustre: Skipped 2 previous similar messages
[ 2764.520126] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2787.839545] LustreError: lustre-MDT0000-mdc-ffff9ad046ce7000: operation mds_reint to node 192.168.203.133@tcp failed: rc = -19
[ 2787.852913] LustreError: Skipped 7 previous similar messages
[ 2807.263266] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 192.168.203.133@tcp) was lost; in progress operations using this service will fail
[ 2807.287132] LustreError: Skipped 3 previous similar messages
[ 2807.331505] Lustre: MGC192.168.203.133@tcp: Connection restored to 192.168.203.133@tcp (at 192.168.203.133@tcp)
[ 2807.341138] Lustre: Skipped 18 previous similar messages
[ 2814.628649] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2842.917693] Lustre: lustre-MDT0000-mdc-ffff9ad046ce7000: Connection to lustre-MDT0000 (at 192.168.203.133@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2842.933507] Lustre: Skipped 10 previous similar messages
[ 2872.890526] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2935.487123] Lustre: 15945:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.133@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2971.025818] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 02:49:55 (1788331795)
[ 3034.321722] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3036.315218] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3316.520700] LustreError: lustre-OST0000-osc-ffff9ad046ce7000: operation ost_write to node 192.168.203.133@tcp failed: rc = -19
[ 3316.528524] LustreError: Skipped 3 previous similar messages
[ 3362.934346] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3365.042446] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3649.024202] Lustre: lustre-OST0000-osc-ffff9ad046ce7000: Connection to lustre-OST0000 (at 192.168.203.133@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3649.040918] Lustre: Skipped 3 previous similar messages
[ 3673.165497] Lustre: lustre-OST0000-osc-ffff9ad046ce7000: Connection restored to 192.168.203.133@tcp (at 192.168.203.133@tcp)
[ 3673.179118] Lustre: Skipped 7 previous similar messages
[ 3690.574549] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3692.733846] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3944.756229] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 03:06:09 (1788332769)
[ 3961.823521] Lustre: 50611:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788332772/real 1788332772]  req@ffff9ad04686f100 x1875198799195520/t0(0) o101->lustre-MDT0000-mdc-ffff9ad046ce7000@192.168.203.133@tcp:12/10 lens 576/1152 e 0 to 1 dl 1788332788 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3961.883742] Lustre: 50611:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
[ 3969.163177] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 03:06:34 (1788332794)
[ 3992.869464] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 03:06:57 (1788332817)
[ 4017.772838] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 03:07:22 (1788332842)
[ 4018.165067] Lustre: Mounted lustre-client - version 2.17.57_104_g6cc0f2c
[ 4048.863362] Lustre: 2378:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788332859/real 1788332859]  req@ffff9ad046964380 x1875198799214208/t0(0) o400->lustre-MDT0000-mdc-ffff9ad046ce7000@192.168.203.133@tcp:12/10 lens 224/224 e 0 to 1 dl 1788332875 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4048.878307] Lustre: 2378:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 4048.885309] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 192.168.203.133@tcp) was lost; in progress operations using this service will fail
[ 4048.892052] LustreError: Skipped 2 previous similar messages
[ 4059.129451] Lustre: Evicted from MGS (at 192.168.203.133@tcp) after server handle changed from 0xfcfbceb164959348 to 0xfcfbceb164aac751
[ 4059.137576] Lustre: Skipped 3 previous similar messages
[ 4065.537459] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4067.297574] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4069.274916] Lustre: Unmounted lustre-client
[ 4075.037891] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 03:08:20 (1788332900)
[ 4211.679218] Lustre: 2378:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788333021/real 1788333021]  req@ffff9ad046964a80 x1875198799246848/t0(0) o4->lustre-OST0000-osc-ffff9ad046ce7000@192.168.203.133@tcp:6/4 lens 488/448 e 0 to 1 dl 1788333037 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4211.717127] Lustre: 2378:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 67 previous similar messages
[ 4258.783258] Lustre: lustre-OST0000-osc-ffff9ad046ce7000: Connection to lustre-OST0000 (at 192.168.203.133@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4258.801799] Lustre: Skipped 15 previous similar messages
[ 4275.193400] Lustre: lustre-OST0000-osc-ffff9ad046ce7000: Connection restored to 192.168.203.133@tcp (at 192.168.203.133@tcp)
[ 4275.206861] Lustre: Skipped 17 previous similar messages
[ 4369.022083] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 03:13:13 (1788333193)
[ 4418.339281] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 03:14:03 (1788333243)
[ 4421.539725] Lustre: Unmounted lustre-client
[ 4444.170896] Lustre: Mounted lustre-client - version 2.17.57_104_g6cc0f2c
[ 4450.805504] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 03:14:35 (1788333275)
[ 4451.108832] LustreError: 56531:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4452.169233] LustreError: 56531:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4452.343700] Lustre: *** cfs_fail_loc=305, val=0***
[ 4474.775752] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 03:14:59 (1788333299)
[ 4475.164177] Lustre: Mounted lustre-client - version 2.17.57_104_g6cc0f2c
[ 4477.508402] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4487.833847] Lustre: Unmounted lustre-client
[ 4494.983163] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 03:15:19 (1788333319)
[ 4586.466139] LustreError: lustre-MDT0000-mdc-ffff9ad0456cf800: operation mds_reint to node 192.168.203.133@tcp failed: rc = -19
[ 4586.472274] LustreError: Skipped 5 previous similar messages
[ 4604.383237] Lustre: 2376:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788333414/real 1788333414]  req@ffff9ad04b7a8e00 x1875198801921536/t0(0) o400->MGC192.168.203.133@tcp@192.168.203.133@tcp:26/25 lens 224/224 e 0 to 1 dl 1788333430 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4604.436317] Lustre: 2376:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 60 previous similar messages
[ 4604.456637] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 192.168.203.133@tcp) was lost; in progress operations using this service will fail
[ 4614.694137] Lustre: Evicted from MGS (at 192.168.203.133@tcp) after server handle changed from 0xfcfbceb164aadf45 to 0xfcfbceb164ad9200
[ 4616.234201] Lustre: 55930:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.133@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 4777.785309] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 03:20:02 (1788333602)
[ 4783.192032] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4796.905644] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 192.168.203.133@tcp) was lost; in progress operations using this service will fail
[ 4796.941720] Lustre: Evicted from MGS (at 192.168.203.133@tcp) after server handle changed from 0xfcfbceb164ad9200 to 0xfcfbceb164b25eb1
[ 4807.263629] LustreError: lustre-MDT0000-mdc-ffff9ad0456cf800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4815.955173] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 03:20:40 (1788333640)
[ 4816.616452] Lustre: Mounted lustre-client - version 2.17.57_104_g6cc0f2c
[ 4823.790220] LustreError: lustre-OST0000-osc-ffff9ad0456cf800: operation ldlm_enqueue to node 192.168.203.133@tcp failed: rc = -107
[ 4823.812074] LustreError: lustre-OST0000-osc-ffff9ad0456cf800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4823.830343] LustreError: 60252:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff9ad0456cf800: namespace resource [0x240000400:0x3e83:0x0].0x0 (ffff9ad04b7d6900) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4835.837224] Lustre: Unmounted lustre-client
[ 4843.740777] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 03:21:08 (1788333668)
[ 4850.159739] LustreError: lustre-MDT0000-mdc-ffff9ad0456cf800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4850.169260] LustreError: 60923:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4850.174904] LustreError: 61007:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9ad0456cf800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff9ad050502900) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4857.757812] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 03:21:22 (1788333682)
[ 4858.076905] LustreError: 61627:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4863.456099] LustreError: 61627:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=0
[ 4863.463574] LustreError: 61627:0:(import.c:711:ptlrpc_connect_import_locked()) already connected
[ 4865.040182] Lustre: lustre-MDT0000-mdc-ffff9ad0456cf800: Connection to lustre-MDT0000 (at 192.168.203.133@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4865.068602] Lustre: Skipped 16 previous similar messages
[ 4865.083674] LustreError: lustre-MDT0000-mdc-ffff9ad0456cf800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4865.091629] LustreError: 61645:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4865.142361] LustreError: 61650:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4866.245268] LustreError: 61656:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4866.258905] LustreError: 61656:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4868.565941] LustreError: 61678:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4868.584494] LustreError: 61678:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4873.359667] LustreError: 61723:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4873.390653] LustreError: 61723:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4875.239719] Lustre: lustre-MDT0000-mdc-ffff9ad0456cf800: Connection restored to 192.168.203.133@tcp (at 192.168.203.133@tcp)
[ 4875.249805] Lustre: Skipped 16 previous similar messages
[ 4882.419738] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 03:21:47 (1788333707)
[ 4922.421749] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4924.184666] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4934.549262] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 03:22:39 (1788333759)
[ 4973.197767] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4974.844992] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4986.050940] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 03:23:30 (1788333810)
[ 5049.291204] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5050.789918] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5059.181609] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 03:24:44 (1788333884)
[ 5094.711329] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5096.182833] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5101.616171] Lustre: Unmounted lustre-client
[ 5152.851719] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5155.273430] Lustre: Mounted lustre-client - version 2.17.57_104_g6cc0f2c
[ 5161.270104] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 03:26:25 (1788333985)
[ 5163.563894] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5165.227898] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 03:26:30 (1788333990)
[ 5194.122093] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 03:26:59 (1788334019)
[ 5195.187790] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5196.602775] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 03:27:01 (1788334021)
[ 5197.653952] Lustre: *** cfs_fail_loc=805, val=0***
[ 5197.687112] Lustre: Mounted lustre-client - version 2.17.57_104_g6cc0f2c
[ 5202.965707] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5223.403064] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 192.168.203.133@tcp) was lost; in progress operations using this service will fail
[ 5223.433555] Lustre: Evicted from MGS (at 192.168.203.133@tcp) after server handle changed from 0xfcfbceb164b26954 to 0xfcfbceb164b26c6b
[ 5224.415775] Lustre: 2377:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788334034/real 1788334034]  req@ffff9ad048e9df80 x1875198803976064/t0(0) o400->lustre-MDT0000-mdc-ffff9ad049e2c800@192.168.203.133@tcp:12/10 lens 224/224 e 0 to 1 dl 1788334050 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5224.460847] Lustre: 2377:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5231.447687] Lustre: Unmounted lustre-client
[ 5238.727524] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 03:27:43 (1788334063)
[ 5281.806961] LustreError: 2375:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff9ad04b7aa680 x1875198803986816/t94489280516(94489280516) o101->lustre-MDT0000-mdc-ffff9ad0506f2000@192.168.203.133@tcp:12/10 lens 664/608 e 0 to 0 dl 1788334124 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5286.869280] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5288.319603] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5296.482476] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 03:28:41 (1788334121)
[ 5297.904518] LustreError: lustre-OST0000-osc-ffff9ad0506f2000: operation ost_write to node 192.168.203.133@tcp failed: rc = -107
[ 5297.911838] LustreError: Skipped 2 previous similar messages
[ 5297.940279] LustreError: lustre-OST0000-osc-ffff9ad0506f2000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5297.955232] Lustre: 2378:0:(llite_lib.c:4344:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.133@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5297.994211] LustreError: 72821:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff9ad0506f2000: namespace resource [0x240000400:0x3ee2:0x0].0x0 (ffff9ad045dfe300) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5305.780603] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 03:28:50 (1788334130)
[ 5307.334955] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5308.802317] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 03:28:53 (1788334133)
[ 5310.327246] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5312.048127] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 03:28:56 (1788334136)
[ 5313.500885] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5315.154924] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 03:29:00 (1788334140)
[ 5316.537586] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5318.396343] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 03:29:03 (1788334143)
[ 5319.959468] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5321.864430] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 03:29:06 (1788334146)
[ 5323.201734] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5325.015472] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 03:29:09 (1788334149)
[ 5326.417500] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5328.421855] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 03:29:13 (1788334153)
[ 5330.179967] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5331.996487] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 03:29:16 (1788334156)
[ 5333.554506] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5335.454505] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 03:29:20 (1788334160)
[ 5337.046766] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5339.129996] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 03:29:23 (1788334163)
[ 5340.892984] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5342.891744] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 03:29:27 (1788334167)
[ 5345.506955] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5347.272968] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 03:29:32 (1788334172)
[ 5368.804687] Lustre: 68284:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.133@tcp: IR log lustre-cliir failed, not fatal: rc = -5
[ 5378.734767] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 03:30:03 (1788334203)
[ 5407.463558] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 03:30:32 (1788334232)
[ 5408.064584] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5417.622271] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 03:30:42 (1788334242)
[ 5417.962451] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5428.652378] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 03:30:53 (1788334253)
[ 5429.041138] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5436.560756] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 03:31:01 (1788334261)
[ 5436.845447] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5444.696059] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 03:31:09 (1788334269)
[ 5445.075388] Lustre: *** cfs_fail_loc=510, val=3***
[ 5453.041410] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 03:31:18 (1788334278)
[ 5453.356915] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5463.131706] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 03:31:28 (1788334288)
[ 5463.469576] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5523.423242] Lustre: lustre-OST0001-osc-ffff9ad0506f2000: Connection to lustre-OST0001 (at 192.168.203.133@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5523.460373] Lustre: Skipped 10 previous similar messages
[ 5530.921179] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 03:32:35 (1788334355)
[ 5531.445129] LustreError: 83028:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5534.436608] LustreError: lustre-MDT0000-mdc-ffff9ad0506f2000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5534.446525] LustreError: 83043:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9ad0506f2000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9ad050417500) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5534.468350] Lustre: lustre-MDT0000-mdc-ffff9ad0506f2000: Connection restored to 192.168.203.133@tcp (at 192.168.203.133@tcp)
[ 5534.475862] Lustre: Skipped 12 previous similar messages
[ 5535.519192] LustreError: 83028:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5537.634283] LustreError: 83058:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5540.512587] LustreError: lustre-MDT0000-mdc-ffff9ad0506f2000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5540.536838] LustreError: 83073:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9ad0506f2000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9ad04b7d6700) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5540.577861] LustreError: 83073:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5541.639227] LustreError: 83058:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5548.720552] LustreError: lustre-MDT0000-mdc-ffff9ad0506f2000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5549.252201] LustreError: 83114:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5552.126709] LustreError: lustre-MDT0000-mdc-ffff9ad0506f2000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5552.189477] LustreError: 83134:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5552.200110] LustreError: 83134:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5553.296680] LustreError: 83140:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5553.314392] LustreError: 83140:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5553.319126] LustreError: 83114:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5553.361340] LustreError: 83114:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9ad0506f2000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5555.662763] LustreError: 83162:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5555.678677] LustreError: 83162:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5559.303678] LustreError: 83189:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5562.234470] LustreError: lustre-MDT0000-mdc-ffff9ad0506f2000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5562.262050] LustreError: 83204:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9ad0506f2000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9ad04b7d6500) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5562.302881] LustreError: 83204:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5563.399231] LustreError: 83189:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5570.297373] LustreError: lustre-MDT0000-mdc-ffff9ad0506f2000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5570.648082] LustreError: 83245:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5573.607484] LustreError: lustre-MDT0000-mdc-ffff9ad0506f2000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5574.722334] LustreError: 83245:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5577.851668] LustreError: lustre-MDT0000-mdc-ffff9ad0506f2000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5577.971222] LustreError: 83293:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5577.980588] LustreError: 83293:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5578.935704] LustreError: 83273:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9ad0506f2000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5587.083668] LustreError: 83345:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5587.090026] LustreError: 83345:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5590.341195] LustreError: lustre-MDT0000-mdc-ffff9ad0506f2000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5590.428837] LustreError: 83367:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5590.437781] LustreError: 83367:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5591.151190] LustreError: 83345:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5591.160149] LustreError: 83345:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5591.168206] LustreError: 83345:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9ad0506f2000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5601.521991] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 03:33:46 (1788334426)
[ 5621.308120] LustreError: 68280:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5621.319533] LustreError: 68280:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5625.391097] LustreError: 68280:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5625.396575] LustreError: 68280:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5632.757867] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 03:34:17 (1788334457)
[ 5681.461814] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 03:35:06 (1788334506)
[ 5746.157978] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 03:36:10 (1788334570)
[ 5779.644846] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 03:36:44 (1788334604)
[ 5780.165772] Lustre: Mounted lustre-client - version 2.17.57_104_g6cc0f2c
[ 5903.522506] Lustre: Unmounted lustre-client
[ 5909.545658] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 03:38:54 (1788334734)
[ 5910.162504] LustreError: 87671:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5912.262968] LustreError: lustre-OST0000-osc-ffff9ad0506f2000: operation ldlm_enqueue to node 192.168.203.133@tcp failed: rc = -107
[ 5912.271310] LustreError: Skipped 9 previous similar messages
[ 5912.290677] LustreError: lustre-OST0000-osc-ffff9ad0506f2000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5912.305056] LustreError: 87754:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff9ad0506f2000: namespace resource [0x240000400:0x3f0d:0x0].0x0 (ffff9ad0522df500) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5912.332341] LustreError: 87754:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 4 previous similar messages
[ 5912.602815] LustreError: 87671:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5912.610188] Lustre: 2377:0:(llite_lib.c:4344:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.133@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5918.558233] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 03:39:03 (1788334743)
[ 5976.031140] Lustre: 88339:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788334747/real 1788334747]  req@ffff9ad050429880 x1875198804158848/t0(0) o101->lustre-MDT0000-mdc-ffff9ad0506f2000@192.168.203.133@tcp:12/10 lens 328/344 e 0 to 1 dl 1788334802 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 5976.055759] Lustre: 88339:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 14 previous similar messages
[ 5982.131259] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 03:40:07 (1788334807)
[ 5983.504571] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 5984.948938] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 03:40:10 (1788334810)
[ 6047.788949] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6049.369389] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 03:41:14 (1788334874)
[ 6107.584107] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 03:42:12 (1788334932)
[ 6109.841665] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6111.427415] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 03:42:16 (1788334936)
[ 6112.746857] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6114.214948] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 03:42:19 (1788334939)
[ 6146.678569] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 03:42:51 (1788334971)
[ 6156.825090] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6182.879255] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 192.168.203.133@tcp) was lost; in progress operations using this service will fail
[ 6182.889135] LustreError: Skipped 3 previous similar messages
[ 6182.897170] Lustre: Evicted from MGS (at 192.168.203.133@tcp) after server handle changed from 0xfcfbceb164b277bd to 0xfcfbceb164b28fe9
[ 6182.907983] Lustre: Skipped 3 previous similar messages
[ 6182.914539] Lustre: MGC192.168.203.133@tcp: Connection restored to 192.168.203.133@tcp (at 192.168.203.133@tcp)
[ 6182.931867] Lustre: Skipped 14 previous similar messages
[ 6184.682724] Lustre: 68284:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.133@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6189.023475] Lustre: lustre-MDT0000-mdc-ffff9ad0506f2000: Connection to lustre-MDT0000 (at 192.168.203.133@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6189.037771] Lustre: Skipped 16 previous similar messages
[ 6204.349880] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6205.645500] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6214.936516] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 03:43:59 (1788335039)
[ 6217.191207] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6218.485588] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 03:44:03 (1788335043)
[ 6229.991619] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 192.168.203.133@tcp) was lost; in progress operations using this service will fail
[ 6230.043761] Lustre: Evicted from MGS (at 192.168.203.133@tcp) after server handle changed from 0xfcfbceb164b28fe9 to 0xfcfbceb164b29426
[ 6242.167283] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 03:44:27 (1788335067)
[ 6280.512190] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 03:45:05 (1788335105)
[ 6327.926637] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6330.639408] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6415.647222] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 192.168.203.133@tcp) was lost; in progress operations using this service will fail
[ 6415.662237] LustreError: Skipped 1 previous similar message
[ 6415.685864] Lustre: Evicted from MGS (at 192.168.203.133@tcp) after server handle changed from 0xfcfbceb164b29a00 to 0xfcfbceb164b2a616
[ 6415.699619] Lustre: Skipped 1 previous similar message
[ 6427.537933] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6428.565811] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6460.226287] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6461.406251] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6567.240556] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 03:49:52 (1788335392)
[ 6575.361397] LustreError: lustre-OST0000-osc-ffff9ad0506f2000: operation ost_setattr to node 192.168.203.133@tcp failed: rc = -19
[ 6575.367296] LustreError: Skipped 41 previous similar messages
[ 6614.135300] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6616.282265] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6825.313129] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 03:54:10 (1788335650)
[ 6928.358163] Lustre: lustre-MDT0000-mdc-ffff9ad0506f2000: Connection to lustre-MDT0000 (at 192.168.203.133@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6928.371033] Lustre: Skipped 6 previous similar messages
[ 6938.597741] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 192.168.203.133@tcp) was lost; in progress operations using this service will fail
[ 6938.609244] LustreError: Skipped 1 previous similar message
[ 6938.622986] Lustre: Evicted from MGS (at 192.168.203.133@tcp) after server handle changed from 0xfcfbceb164b2aaa0 to 0xfcfbceb164c24d18
[ 6938.632531] Lustre: Skipped 1 previous similar message
[ 6938.637766] Lustre: MGC192.168.203.133@tcp: Connection restored to 192.168.203.133@tcp (at 192.168.203.133@tcp)
[ 6938.642958] Lustre: Skipped 9 previous similar messages
[ 6972.440797] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 03:56:37 (1788335797)
[ 6973.290389] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 6974.378840] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 03:56:39 (1788335799)
[ 6975.571826] LustreError: lustre-MDT0000-mdc-ffff9ad0506f2000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6975.585792] LustreError: 101985:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9ad0506f2000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff9ad071cf3f00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6975.595471] LustreError: 101985:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 6980.171886] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 03:56:45 (1788335805)
[ 7146.088850] LustreError: lustre-OST0000-osc-ffff9ad0506f2000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7147.411890] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 03:59:32 (1788335972)
[ 7172.063227] Lustre: 2378:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788335978/real 1788335978]  req@ffff9ad0467a3b80 x1875198821895424/t0(0) o4->lustre-OST0000-osc-ffff9ad0506f2000@192.168.203.133@tcp:6/4 lens 4584/448 e 0 to 1 dl 1788335998 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7172.086750] Lustre: 2378:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 23 previous similar messages
[ 7187.398977] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 04:00:12 (1788336012)
[ 7188.689813] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7189.800860] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 04:00:15 (1788336015)
[ 7190.809242] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7191.873728] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 04:00:17 (1788336017)
[ 7223.121760] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 04:00:48 (1788336048)
[ 7259.122773] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 192.168.203.133@tcp) was lost; in progress operations using this service will fail
[ 7259.155599] Lustre: Evicted from MGS (at 192.168.203.133@tcp) after server handle changed from 0xfcfbceb164c24d18 to 0xfcfbceb164c27478
[ 7259.165454] Lustre: Skipped 1 previous similar message
[ 7270.243035] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 04:01:35 (1788336095)
[ 7271.307848] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7272.305165] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 04:01:37 (1788336097)
[ 7273.361948] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7274.555531] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 04:01:39 (1788336099)
[ 7281.495362] Lustre: Unmounted lustre-client
[ 7282.454435] Lustre: Mounted lustre-client - version 2.17.57_104_g6cc0f2c
[ 7284.664483] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7308.320280] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2