[ 1926.666502] LustreError: lustre-OST0000-osc-ffff94d7d1615000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1926.696420] Lustre: 2356:0:(llite_lib.c:4202:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.120@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1933.861524] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 11:59:32 (1782230372)
[ 1936.105236] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1937.676326] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 11:59:36 (1782230376)
[ 1939.502988] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1941.231625] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 11:59:39 (1782230379)
[ 1944.541551] LustreError: lustre-MDT0000-mdc-ffff94d7d1615000: operation ldlm_enqueue to node 192.168.204.120@tcp failed: rc = -19
[ 1964.385086] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 192.168.204.120@tcp) was lost; in progress operations using this service will fail
[ 1964.425164] Lustre: Evicted from MGS (at 192.168.204.120@tcp) after server handle changed from 0x3f13028abada4fdb to 0x3f13028abada6a1b
[ 2064.287519] LustreError: lustre-MDT0000-mdc-ffff94d7d1615000: operation mds_close to node 192.168.204.120@tcp failed: rc = -19
[ 2064.308241] LustreError: Skipped 1 previous similar message
[ 2081.055421] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 192.168.204.120@tcp) was lost; in progress operations using this service will fail
[ 2091.380904] Lustre: Evicted from MGS (at 192.168.204.120@tcp) after server handle changed from 0x3f13028abada6a1b to 0x3f13028abadc1b34
[ 2112.194969] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 12:02:30 (1782230550)
[ 2112.745144] Lustre: *** cfs_fail_loc=305, val=0***
[ 2112.749063] Lustre: Skipped 2 previous similar messages
[ 2132.639253] LustreError: lustre-OST0001-osc-ffff94d7d1615000: operation ost_connect to node 192.168.204.120@tcp failed: rc = -75
[ 2158.580716] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 192.168.204.120@tcp) was lost; in progress operations using this service will fail
[ 2158.611374] Lustre: Evicted from MGS (at 192.168.204.120@tcp) after server handle changed from 0x3f13028abadc1b34 to 0x3f13028abadc2185
[ 2160.067973] Lustre: lustre-MDT0000-mdc-ffff94d7d1615000: Connection restored to 192.168.204.120@tcp (at 192.168.204.120@tcp)
[ 2160.082415] Lustre: Skipped 14 previous similar messages
[ 2174.232194] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2176.236741] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2185.932705] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 12:03:44 (1782230624)
[ 2199.534346] Lustre: lustre-MDT0000-mdc-ffff94d7d1615000: Connection to lustre-MDT0000 (at 192.168.204.120@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2199.542539] Lustre: Skipped 12 previous similar messages
[ 2199.545721] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 192.168.204.120@tcp) was lost; in progress operations using this service will fail
[ 2199.562146] Lustre: Evicted from MGS (at 192.168.204.120@tcp) after server handle changed from 0x3f13028abadc2185 to 0x3f13028abadc2686
[ 2199.624177] LustreError: lustre-MDT0000-mdc-ffff94d7d1615000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2223.202593] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 12:04:21 (1782230661)
[ 2237.010237] LustreError: lustre-OST0000-osc-ffff94d7d1615000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2256.348362] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 12:04:54 (1782230694)
[ 2269.996196] LustreError: lustre-MDT0000-mdc-ffff94d7d1615000: operation mds_reint to node 192.168.204.120@tcp failed: rc = -19
[ 2287.583252] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 192.168.204.120@tcp) was lost; in progress operations using this service will fail
[ 2297.769606] Lustre: Evicted from MGS (at 192.168.204.120@tcp) after server handle changed from 0x3f13028abadc2686 to 0x3f13028abadc6f9d
[ 2313.958803] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2316.109302] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2399.199200] Lustre: 2353:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782230823/real 1782230823]  req@ffff94d7c35b0000 x1868802047741440/t0(0) o400->MGC192.168.204.120@tcp@192.168.204.120@tcp:26/25 lens 224/224 e 0 to 1 dl 1782230839 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2399.236425] Lustre: 2353:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 12 previous similar messages
[ 2399.243505] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 192.168.204.120@tcp) was lost; in progress operations using this service will fail
[ 2409.457431] Lustre: Evicted from MGS (at 192.168.204.120@tcp) after server handle changed from 0x3f13028abadc6f9d to 0x3f13028abade095c
[ 2428.938774] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2431.683652] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2515.965278] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 192.168.204.120@tcp) was lost; in progress operations using this service will fail
[ 2515.987402] Lustre: Evicted from MGS (at 192.168.204.120@tcp) after server handle changed from 0x3f13028abade095c to 0x3f13028abadfb018
[ 2534.048234] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2536.429833] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2566.378426] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 12:10:05 (1782231005)
[ 2570.674085] LustreError: lustre-MDT0000-mdc-ffff94d7d1615000: operation mds_reint to node 192.168.204.120@tcp failed: rc = -19
[ 2570.686058] LustreError: Skipped 5 previous similar messages
[ 2588.640550] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 192.168.204.120@tcp) was lost; in progress operations using this service will fail
[ 2588.681688] Lustre: Evicted from MGS (at 192.168.204.120@tcp) after server handle changed from 0x3f13028abadfb018 to 0x3f13028abae0a064
[ 2597.506211] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2629.053586] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2664.074610] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2703.931898] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2749.983409] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 192.168.204.120@tcp) was lost; in progress operations using this service will fail
[ 2750.000426] LustreError: Skipped 3 previous similar messages
[ 2750.014294] Lustre: Evicted from MGS (at 192.168.204.120@tcp) after server handle changed from 0x3f13028abae11d0d to 0x3f13028abae164d4
[ 2750.030946] Lustre: Skipped 3 previous similar messages
[ 2751.919310] Lustre: 15905:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.204.120@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2768.385498] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2768.580623] Lustre: lustre-MDT0000-mdc-ffff94d7d1615000: Connection restored to 192.168.204.120@tcp (at 192.168.204.120@tcp)
[ 2768.603453] Lustre: Skipped 17 previous similar messages
[ 2835.503015] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2868.510861] Lustre: lustre-MDT0000-mdc-ffff94d7d1615000: Connection to lustre-MDT0000 (at 192.168.204.120@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2868.529084] Lustre: Skipped 9 previous similar messages
[ 2922.133408] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 12:16:00 (1782231360)
[ 2982.031774] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2985.331047] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3268.259525] LustreError: lustre-OST0000-osc-ffff94d7d1615000: operation ldlm_enqueue to node 192.168.204.120@tcp failed: rc = -19
[ 3268.266400] LustreError: Skipped 8 previous similar messages
[ 3312.926378] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3315.632280] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3600.415385] Lustre: lustre-OST0000-osc-ffff94d7d1615000: Connection to lustre-OST0000 (at 192.168.204.120@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3600.433423] Lustre: Skipped 2 previous similar messages
[ 3637.653852] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3639.643937] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3894.642580] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 12:32:13 (1782232333)
[ 3911.135225] Lustre: 50411:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782232335/real 1782232335]  req@ffff94d7c3656300 x1868802068705152/t0(0) o101->lustre-MDT0000-mdc-ffff94d7d1615000@192.168.204.120@tcp:12/10 lens 576/1152 e 0 to 1 dl 1782232351 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3911.166781] Lustre: 50411:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
[ 3911.196458] Lustre: lustre-MDT0000-mdc-ffff94d7d1615000: Connection restored to 192.168.204.120@tcp (at 192.168.204.120@tcp)
[ 3911.212335] Lustre: Skipped 5 previous similar messages
[ 3917.536438] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 12:32:36 (1782232356)
[ 3940.468539] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 12:32:59 (1782232379)
[ 3963.114622] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 12:33:22 (1782232402)
[ 3963.483611] Lustre: Mounted lustre-client
[ 3994.084685] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 192.168.204.120@tcp) was lost; in progress operations using this service will fail
[ 3994.095954] LustreError: Skipped 2 previous similar messages
[ 3994.110088] Lustre: Evicted from MGS (at 192.168.204.120@tcp) after server handle changed from 0x3f13028abae2a602 to 0x3f13028abaf680a3
[ 3994.115467] Lustre: Skipped 2 previous similar messages
[ 3995.103204] Lustre: 2354:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782232418/real 1782232418]  req@ffff94d7c4924700 x1868802068724096/t0(0) o400->lustre-MDT0000-mdc-ffff94d7d151e800@192.168.204.120@tcp:12/10 lens 224/224 e 0 to 1 dl 1782232434 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3995.122107] Lustre: 2354:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 4002.174667] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4003.310423] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4005.045668] Lustre: Unmounted lustre-client
[ 4009.845327] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 12:34:08 (1782232448)
[ 4160.991127] Lustre: 2356:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782232584/real 1782232584]  req@ffff94d7c4857100 x1868802068755840/t0(0) o4->lustre-OST0000-osc-ffff94d7d1615000@192.168.204.120@tcp:6/4 lens 488/448 e 0 to 1 dl 1782232600 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4161.708501] Lustre: 2356:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 78 previous similar messages
[ 4209.631738] Lustre: lustre-OST0000-osc-ffff94d7d1615000: Connection to lustre-OST0000 (at 192.168.204.120@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4209.674990] Lustre: Skipped 16 previous similar messages
[ 4304.731238] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 12:39:03 (1782232743)
[ 4354.287187] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 12:39:52 (1782232792)
[ 4358.146419] Lustre: Unmounted lustre-client
[ 4381.100811] Lustre: Mounted lustre-client
[ 4387.830946] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 12:40:26 (1782232826)
[ 4388.168451] LustreError: 56318:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4389.215150] LustreError: 56318:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4389.507313] Lustre: *** cfs_fail_loc=305, val=0***
[ 4413.637314] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 12:40:51 (1782232851)
[ 4414.182781] Lustre: Mounted lustre-client
[ 4417.350899] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4427.708620] Lustre: Unmounted lustre-client
[ 4435.274327] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 12:41:13 (1782232873)
[ 4540.331775] LustreError: lustre-MDT0000-mdc-ffff94d7c364a800: operation mds_reint to node 192.168.204.120@tcp failed: rc = -19
[ 4540.335937] LustreError: Skipped 1 previous similar message
[ 4559.583550] Lustre: 2355:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782232983/real 1782232983]  req@ffff94d7c5ce9500 x1868802071421696/t0(0) o400->MGC192.168.204.120@tcp@192.168.204.120@tcp:26/25 lens 224/224 e 0 to 1 dl 1782232999 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4559.638085] Lustre: 2355:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 55 previous similar messages
[ 4559.648267] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 192.168.204.120@tcp) was lost; in progress operations using this service will fail
[ 4569.829995] Lustre: Evicted from MGS (at 192.168.204.120@tcp) after server handle changed from 0x3f13028abaf6a67b to 0x3f13028abaf95562
[ 4569.855807] Lustre: MGC192.168.204.120@tcp: Connection restored to 192.168.204.120@tcp (at 192.168.204.120@tcp)
[ 4569.864759] Lustre: Skipped 23 previous similar messages
[ 4749.112727] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 12:46:27 (1782233187)
[ 4755.357862] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4773.871038] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 192.168.204.120@tcp) was lost; in progress operations using this service will fail
[ 4773.908582] Lustre: Evicted from MGS (at 192.168.204.120@tcp) after server handle changed from 0x3f13028abaf95562 to 0x3f13028abafe2a39
[ 4773.932040] LustreError: lustre-MDT0000-mdc-ffff94d7c364a800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4789.983938] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 12:47:08 (1782233228)
[ 4790.540034] Lustre: Mounted lustre-client
[ 4798.218204] LustreError: lustre-OST0000-osc-ffff94d7c364a800: operation ldlm_enqueue to node 192.168.204.120@tcp failed: rc = -107
[ 4798.242745] LustreError: lustre-OST0000-osc-ffff94d7c364a800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4809.906067] Lustre: Unmounted lustre-client
[ 4819.335607] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 12:47:36 (1782233256)
[ 4823.082179] Lustre: lustre-MDT0000-mdc-ffff94d7c364a800: Connection to lustre-MDT0000 (at 192.168.204.120@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4823.108875] Lustre: Skipped 12 previous similar messages
[ 4826.187841] LustreError: lustre-MDT0000-mdc-ffff94d7c364a800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4826.202950] LustreError: 60700:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4834.805394] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 12:47:52 (1782233272)
[ 4835.023607] LustreError: 61401:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4839.215863] LustreError: lustre-MDT0000-mdc-ffff94d7c364a800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4839.247499] LustreError: 61420:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4839.270039] LustreError: 61401:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=765
[ 4839.286921] LustreError: 61401:0:(import.c:719:ptlrpc_connect_import_locked()) already connecting
[ 4839.369446] LustreError: 61425:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4840.520427] LustreError: 61431:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4840.537463] LustreError: 61431:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4843.039810] LustreError: 61454:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4843.068647] LustreError: 61454:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4847.900176] LustreError: 61499:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4847.912284] LustreError: 61499:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4859.414459] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 12:48:17 (1782233297)
[ 4903.440933] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4905.235247] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4916.698740] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 12:49:14 (1782233354)
[ 4956.025432] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4957.973487] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4971.551621] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 12:50:09 (1782233409)
[ 5036.414778] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5038.003794] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5047.671202] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 12:51:25 (1782233485)
[ 5086.532273] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5088.104721] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5095.097693] Lustre: Unmounted lustre-client
[ 5151.310533] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5154.183526] Lustre: Mounted lustre-client
[ 5161.491784] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 12:53:19 (1782233599)
[ 5164.389641] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5166.506649] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 12:53:24 (1782233604)
[ 5200.030917] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 12:53:58 (1782233638)
[ 5201.812525] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5204.030674] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 12:54:02 (1782233642)
[ 5205.708937] Lustre: *** cfs_fail_loc=805, val=0***
[ 5205.790365] Lustre: Mounted lustre-client
[ 5212.215167] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5232.611360] Lustre: 2354:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782233656/real 1782233656]  req@ffff94d7c5d37480 x1868802073491584/t0(0) o400->lustre-MDT0000-mdc-ffff94d7d151d000@192.168.204.120@tcp:12/10 lens 224/224 e 0 to 1 dl 1782233672 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5232.614689] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 192.168.204.120@tcp) was lost; in progress operations using this service will fail
[ 5232.661288] Lustre: 2354:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
[ 5242.865810] Lustre: Evicted from MGS (at 192.168.204.120@tcp) after server handle changed from 0x3f13028abafe3457 to 0x3f13028abafe3672
[ 5242.924040] Lustre: MGC192.168.204.120@tcp: Connection restored to 192.168.204.120@tcp (at 192.168.204.120@tcp)
[ 5242.933025] Lustre: Skipped 12 previous similar messages
[ 5257.211095] LustreError: 2352:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff94d7d151d000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff94d7c5c84900) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5259.895848] Lustre: Unmounted lustre-client
[ 5268.964872] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 12:55:07 (1782233707)
[ 5295.927276] Lustre: 68022:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.204.120@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 5311.561497] LustreError: 2352:0:(client.c:3447:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff94d7c5c2dc00 x1868802073503360/t90194313221(90194313221) o101->lustre-MDT0000-mdc-ffff94d7d1616000@192.168.204.120@tcp:12/10 lens 648/608 e 0 to 0 dl 1782233767 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5318.765483] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5320.809559] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5330.506933] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 12:56:08 (1782233768)
[ 5331.961143] LustreError: lustre-OST0000-osc-ffff94d7d1616000: operation ost_write to node 192.168.204.120@tcp failed: rc = -107
[ 5331.975904] LustreError: Skipped 2 previous similar messages
[ 5331.994276] LustreError: lustre-OST0000-osc-ffff94d7d1616000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5332.018176] Lustre: 2356:0:(llite_lib.c:4202:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.120@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5332.074203] LustreError: 72545:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff94d7d1616000: namespace resource [0x240000400:0x3ec2:0x0].0x0 (ffff94d7c5d0d500) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5332.100938] LustreError: 72545:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5341.506867] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 12:56:19 (1782233779)
[ 5343.085818] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5345.344644] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 12:56:23 (1782233783)
[ 5347.240851] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5349.029211] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 12:56:27 (1782233787)
[ 5350.806853] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5352.585202] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 12:56:31 (1782233791)
[ 5354.142377] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5356.081763] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 12:56:34 (1782233794)
[ 5357.870437] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5359.569676] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 12:56:38 (1782233798)
[ 5361.207837] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5362.769946] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 12:56:41 (1782233801)
[ 5364.517163] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5366.377281] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 12:56:44 (1782233804)
[ 5367.964395] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5369.521882] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 12:56:48 (1782233808)
[ 5371.031828] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5372.878947] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 12:56:51 (1782233811)
[ 5374.697508] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5376.446940] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 12:56:54 (1782233814)
[ 5377.782183] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5379.358505] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 12:56:58 (1782233818)
[ 5381.882838] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5383.829431] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 12:57:02 (1782233822)
[ 5415.511166] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 12:57:33 (1782233853)
[ 5446.780929] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 12:58:05 (1782233885)
[ 5447.523725] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5458.049730] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 12:58:16 (1782233896)
[ 5458.496669] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5470.238381] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 12:58:28 (1782233908)
[ 5470.884822] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5481.741237] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 12:58:40 (1782233920)
[ 5482.342035] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5492.857359] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 12:58:51 (1782233931)
[ 5493.378284] Lustre: *** cfs_fail_loc=510, val=3***
[ 5503.045096] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 12:59:01 (1782233941)
[ 5503.461142] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5514.804550] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 12:59:13 (1782233953)
[ 5515.310506] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5574.623272] Lustre: lustre-OST0000-osc-ffff94d7d1616000: Connection to lustre-OST0000 (at 192.168.204.120@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5574.645158] Lustre: Skipped 12 previous similar messages
[ 5583.593921] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 13:00:21 (1782234021)
[ 5584.283091] LustreError: 82737:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5587.443673] LustreError: lustre-MDT0000-mdc-ffff94d7d1616000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5587.467963] LustreError: 82752:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff94d7d1616000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff94d7c346fc00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5588.289689] LustreError: 82737:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5590.462754] LustreError: 82769:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5593.633767] LustreError: lustre-MDT0000-mdc-ffff94d7d1616000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5593.662293] LustreError: 82783:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff94d7d1616000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff94d7c5e15d00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5593.701192] LustreError: 82783:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5594.463309] LustreError: 82769:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5601.577541] LustreError: lustre-MDT0000-mdc-ffff94d7d1616000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5601.988132] LustreError: 82824:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5605.084227] LustreError: lustre-MDT0000-mdc-ffff94d7d1616000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5605.160572] LustreError: 82844:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5605.172272] LustreError: 82844:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5606.071259] LustreError: 82824:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5606.084380] LustreError: 82824:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff94d7d1616000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5606.313201] LustreError: 82850:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5606.331032] LustreError: 82850:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5608.643523] LustreError: 82873:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5608.656826] LustreError: 82873:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5612.400287] LustreError: 82899:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5615.458285] LustreError: lustre-MDT0000-mdc-ffff94d7d1616000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5615.473393] LustreError: 82914:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff94d7d1616000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff94d7c5d0d300) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5615.495950] LustreError: 82914:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5616.495836] LustreError: 82899:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5623.731959] LustreError: lustre-MDT0000-mdc-ffff94d7d1616000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5624.154089] LustreError: 82967:0:(ldlm_flock.c:858:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5627.251099] LustreError: lustre-MDT0000-mdc-ffff94d7d1616000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5627.268790] LustreError: 82981:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff94d7d1616000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff94d7c5e15800) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5627.284705] LustreError: 82981:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5628.247134] LustreError: 82967:0:(ldlm_flock.c:858:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5631.405816] LustreError: lustre-MDT0000-mdc-ffff94d7d1616000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5631.516774] LustreError: 83014:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5631.528528] LustreError: 83014:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5632.449800] LustreError: 82995:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff94d7d1616000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5640.631241] LustreError: 83066:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5640.642172] LustreError: 83066:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5644.390556] LustreError: lustre-MDT0000-mdc-ffff94d7d1616000: operation ldlm_enqueue to node 192.168.204.120@tcp failed: rc = -107
[ 5644.399993] LustreError: Skipped 8 previous similar messages
[ 5644.442732] LustreError: lustre-MDT0000-mdc-ffff94d7d1616000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5644.519306] LustreError: 83089:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5644.529749] LustreError: 83089:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5644.719489] LustreError: 83066:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5644.727050] LustreError: 83066:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5644.760527] LustreError: 83066:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff94d7d1616000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5656.773238] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 13:01:35 (1782234095)
[ 5677.074583] LustreError: 79489:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5677.082856] LustreError: 79489:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5681.143128] LustreError: 79489:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5681.154272] LustreError: 79489:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5687.539098] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 13:02:05 (1782234125)
[ 5739.816538] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 13:02:57 (1782234177)
[ 5803.200672] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 13:04:01 (1782234241)
[ 5836.994242] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 13:04:35 (1782234275)
[ 5837.564156] Lustre: Mounted lustre-client
[ 5960.685608] Lustre: Unmounted lustre-client
[ 5967.607759] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 13:06:46 (1782234406)
[ 5968.160090] LustreError: 87392:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5970.788313] LustreError: lustre-OST0000-osc-ffff94d7d1616000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5970.802799] LustreError: 87480:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff94d7d1616000: namespace resource [0x240000400:0x3eed:0x0].0x0 (ffff94d7c346ff00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5970.825878] LustreError: 87480:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5971.119623] LustreError: 87392:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5971.133289] Lustre: 2355:0:(llite_lib.c:4202:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.120@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5971.164625] Lustre: lustre-OST0000-osc-ffff94d7d1616000: Connection restored to 192.168.204.120@tcp (at 192.168.204.120@tcp)
[ 5971.178743] Lustre: Skipped 18 previous similar messages
[ 5978.601269] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 13:06:56 (1782234416)
[ 6036.447528] Lustre: 88060:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782234421/real 1782234421]  req@ffff94d7c5d37100 x1868802073673344/t0(0) o101->lustre-MDT0000-mdc-ffff94d7d1616000@192.168.204.120@tcp:12/10 lens 328/344 e 0 to 1 dl 1782234476 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 6036.468179] Lustre: 88060:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 19 previous similar messages
[ 6042.567638] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 13:08:01 (1782234481)
[ 6044.139641] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 6045.867474] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 13:08:04 (1782234484)
[ 6111.035160] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6113.223546] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 13:09:11 (1782234551)
[ 6171.193356] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 13:10:09 (1782234609)
[ 6173.799855] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6175.955823] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 13:10:14 (1782234614)
[ 6177.716759] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6179.828405] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 13:10:17 (1782234617)
[ 6217.951522] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 13:10:56 (1782234656)
[ 6229.053930] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6256.622571] Lustre: lustre-MDT0000-mdc-ffff94d7d1616000: Connection to lustre-MDT0000 (at 192.168.204.120@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6256.651835] Lustre: Skipped 16 previous similar messages
[ 6256.663562] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 192.168.204.120@tcp) was lost; in progress operations using this service will fail
[ 6256.680727] LustreError: Skipped 2 previous similar messages
[ 6256.697569] Lustre: Evicted from MGS (at 192.168.204.120@tcp) after server handle changed from 0x3f13028abafe42b2 to 0x3f13028abafe5bd3
[ 6256.704811] Lustre: Skipped 2 previous similar messages
[ 6271.833668] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6273.581507] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6284.345645] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 13:12:02 (1782234722)
[ 6286.862524] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6288.496880] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 13:12:07 (1782234727)
[ 6302.710919] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 192.168.204.120@tcp) was lost; in progress operations using this service will fail
[ 6302.745893] Lustre: Evicted from MGS (at 192.168.204.120@tcp) after server handle changed from 0x3f13028abafe5bd3 to 0x3f13028abafe5fae
[ 6315.706789] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 13:12:33 (1782234753)
[ 6340.591761] Lustre: 68022:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.204.120@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6369.054269] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 13:13:27 (1782234807)
[ 6377.481214] LustreError: lustre-OST0000-osc-ffff94d7d1616000: operation ost_setattr to node 192.168.204.120@tcp failed: rc = -107
[ 6377.497430] LustreError: Skipped 1 previous similar message
[ 6440.530204] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 6444.207859] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6533.088361] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 192.168.204.120@tcp) was lost; in progress operations using this service will fail
[ 6533.108535] LustreError: Skipped 1 previous similar message
[ 6533.137760] Lustre: Evicted from MGS (at 192.168.204.120@tcp) after server handle changed from 0x3f13028abafe61bb to 0x3f13028abafe700f
[ 6533.148143] Lustre: Skipped 1 previous similar message
[ 6534.513128] Lustre: 68022:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.204.120@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6559.781655] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6561.964102] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6594.559412] Lustre: MGC192.168.204.120@tcp: Connection restored to 192.168.204.120@tcp (at 192.168.204.120@tcp)
[ 6594.572829] Lustre: Skipped 11 previous similar messages
[ 6609.314114] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6610.787890] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6639.519148] Lustre: 2356:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782235019/real 1782235019]  req@ffff94d7eae9c380 x1868802080204672/t0(0) o400->lustre-MDT0000-mdc-ffff94d7d1616000@192.168.204.120@tcp:12/10 lens 224/224 e 0 to 1 dl 1782235079 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6639.550485] Lustre: 2356:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
[ 6741.219126] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 13:19:39 (1782235179)
[ 6802.537090] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 6805.640453] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 7061.886801] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 13:25:00 (1782235500)
[ 7244.774777] Lustre: lustre-MDT0000-mdc-ffff94d7d1616000: Connection to lustre-MDT0000 (at 192.168.204.120@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7244.786534] Lustre: Skipped 6 previous similar messages
[ 7260.140112] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 192.168.204.120@tcp) was lost; in progress operations using this service will fail
[ 7260.151824] LustreError: Skipped 1 previous similar message
[ 7260.165312] Lustre: Evicted from MGS (at 192.168.204.120@tcp) after server handle changed from 0x3f13028abafe7422 to 0x3f13028abb0de109
[ 7260.170729] Lustre: Skipped 1 previous similar message
[ 7260.178672] Lustre: MGC192.168.204.120@tcp: Connection restored to 192.168.204.120@tcp (at 192.168.204.120@tcp)
[ 7260.184942] Lustre: Skipped 1 previous similar message
[ 7297.339845] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 13:28:55 (1782235735)
[ 7299.500718] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7301.574180] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 13:28:59 (1782235739)
[ 7304.428525] LustreError: lustre-MDT0000-mdc-ffff94d7d1616000: operation ldlm_enqueue to node 192.168.204.120@tcp failed: rc = -107
[ 7304.447707] LustreError: Skipped 91 previous similar messages
[ 7304.458954] LustreError: lustre-MDT0000-mdc-ffff94d7d1616000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7315.114489] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 13:29:13 (1782235753)
[ 7483.184567] LustreError: lustre-OST0000-osc-ffff94d7d1616000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7485.553645] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 13:32:03 (1782235923)
[ 7511.967468] Lustre: 2353:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782235931/real 1782235931]  req@ffff94d7eac1c700 x1868802091126400/t0(0) o4->lustre-OST0000-osc-ffff94d7d1616000@192.168.204.120@tcp:6/4 lens 4584/448 e 0 to 1 dl 1782235951 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7512.008984] Lustre: 2353:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 7530.074895] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 13:32:48 (1782235968)
[ 7531.690345] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7533.684642] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 13:32:52 (1782235972)
[ 7535.223402] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7537.843988] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 13:32:55 (1782235975)
[ 7574.039730] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 13:33:32 (1782236012)
[ 7610.346636] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 192.168.204.120@tcp) was lost; in progress operations using this service will fail
[ 7610.388318] Lustre: Evicted from MGS (at 192.168.204.120@tcp) after server handle changed from 0x3f13028abb0de109 to 0x3f13028abb0e175e
[ 7610.404302] Lustre: Skipped 1 previous similar message
[ 7627.625877] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 13:34:26 (1782236066)
[ 7629.429829] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7631.094389] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 13:34:29 (1782236069)
[ 7632.517960] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7634.374750] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 13:34:32 (1782236072)
[ 7648.914922] Lustre: Unmounted lustre-client
[ 7650.576239] Lustre: Mounted lustre-client
[ 7653.288426] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7682.114404] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2