[ 1981.871557] LustreError: lustre-OST0000-osc-ffff94750c65d000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1981.890443] Lustre: 2373:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.206.158@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1987.727583] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 11:31:35 (1788535895)
[ 1990.614530] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1993.141949] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 11:31:40 (1788535900)
[ 1995.126528] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1997.021138] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 11:31:44 (1788535904)
[ 2000.763746] LustreError: lustre-MDT0000-mdc-ffff94750c65d000: operation mds_reint to node 192.168.206.158@tcp failed: rc = -19
[ 2000.772297] LustreError: Skipped 1 previous similar message
[ 2020.323050] LustreError: MGC192.168.206.158@tcp: Connection to MGS (at 192.168.206.158@tcp) was lost; in progress operations using this service will fail
[ 2030.521952] Lustre: Evicted from MGS (at 192.168.206.158@tcp) after server handle changed from 0xa12e5faa76e3ec8e to 0xa12e5faa76e408bf
[ 2123.806471] LustreError: lustre-MDT0000-mdc-ffff94750c65d000: operation ldlm_enqueue to node 192.168.206.158@tcp failed: rc = -19
[ 2123.814293] LustreError: Skipped 2 previous similar messages
[ 2143.712701] LustreError: MGC192.168.206.158@tcp: Connection to MGS (at 192.168.206.158@tcp) was lost; in progress operations using this service will fail
[ 2143.759794] Lustre: Evicted from MGS (at 192.168.206.158@tcp) after server handle changed from 0xa12e5faa76e408bf to 0xa12e5faa76e5c76f
[ 2145.526099] Lustre: 15961:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.158@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2169.451562] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 11:34:37 (1788536077)
[ 2169.956562] Lustre: *** cfs_fail_loc=305, val=0***
[ 2169.959800] Lustre: Skipped 2 previous similar messages
[ 2189.647279] LustreError: lustre-OST0001-osc-ffff94750c65d000: operation ost_connect to node 192.168.206.158@tcp failed: rc = -75
[ 2215.414837] LustreError: MGC192.168.206.158@tcp: Connection to MGS (at 192.168.206.158@tcp) was lost; in progress operations using this service will fail
[ 2215.448788] Lustre: Evicted from MGS (at 192.168.206.158@tcp) after server handle changed from 0xa12e5faa76e5c76f to 0xa12e5faa76e5cd7a
[ 2217.227484] Lustre: lustre-MDT0000-mdc-ffff94750c65d000: Connection restored to 192.168.206.158@tcp (at 192.168.206.158@tcp)
[ 2217.234101] Lustre: Skipped 14 previous similar messages
[ 2229.640685] Lustre: DEBUG MARKER: oleg658-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2231.375457] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2240.390442] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 11:35:48 (1788536148)
[ 2256.373152] LustreError: MGC192.168.206.158@tcp: Connection to MGS (at 192.168.206.158@tcp) was lost; in progress operations using this service will fail
[ 2256.374505] Lustre: lustre-MDT0000-mdc-ffff94750c65d000: Connection to lustre-MDT0000 (at 192.168.206.158@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2256.430227] Lustre: Skipped 12 previous similar messages
[ 2256.430799] Lustre: Evicted from MGS (at 192.168.206.158@tcp) after server handle changed from 0xa12e5faa76e5cd7a to 0xa12e5faa76e5d354
[ 2256.550984] LustreError: lustre-MDT0000-mdc-ffff94750c65d000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2279.236165] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 11:36:27 (1788536187)
[ 2297.323176] LustreError: lustre-OST0000-osc-ffff94750c65d000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2314.184904] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 11:37:01 (1788536221)
[ 2328.197944] LustreError: lustre-MDT0000-mdc-ffff94750c65d000: operation mds_close to node 192.168.206.158@tcp failed: rc = -19
[ 2348.564702] LustreError: MGC192.168.206.158@tcp: Connection to MGS (at 192.168.206.158@tcp) was lost; in progress operations using this service will fail
[ 2348.595644] Lustre: Evicted from MGS (at 192.168.206.158@tcp) after server handle changed from 0xa12e5faa76e5d354 to 0xa12e5faa76e616bb
[ 2365.275187] Lustre: DEBUG MARKER: oleg658-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2367.225418] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2450.719218] Lustre: 2373:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788536344/real 1788536344]  req@ffff94750508ce00 x1875413841745024/t0(0) o400->MGC192.168.206.158@tcp@192.168.206.158@tcp:26/25 lens 224/224 e 0 to 1 dl 1788536360 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2450.748527] Lustre: 2373:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 12 previous similar messages
[ 2450.756421] LustreError: MGC192.168.206.158@tcp: Connection to MGS (at 192.168.206.158@tcp) was lost; in progress operations using this service will fail
[ 2460.970932] Lustre: Evicted from MGS (at 192.168.206.158@tcp) after server handle changed from 0xa12e5faa76e616bb to 0xa12e5faa76e7cf52
[ 2482.258749] Lustre: DEBUG MARKER: oleg658-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2484.165513] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2568.671208] LustreError: MGC192.168.206.158@tcp: Connection to MGS (at 192.168.206.158@tcp) was lost; in progress operations using this service will fail
[ 2578.925650] Lustre: Evicted from MGS (at 192.168.206.158@tcp) after server handle changed from 0xa12e5faa76e7cf52 to 0xa12e5faa76e94944
[ 2596.164224] Lustre: DEBUG MARKER: oleg658-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2598.513517] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2628.127897] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 11:42:16 (1788536536)
[ 2633.238041] LustreError: lustre-MDT0000-mdc-ffff94750c65d000: operation ldlm_enqueue to node 192.168.206.158@tcp failed: rc = -19
[ 2633.247229] LustreError: Skipped 4 previous similar messages
[ 2652.726579] LustreError: MGC192.168.206.158@tcp: Connection to MGS (at 192.168.206.158@tcp) was lost; in progress operations using this service will fail
[ 2652.762423] Lustre: Evicted from MGS (at 192.168.206.158@tcp) after server handle changed from 0xa12e5faa76e94944 to 0xa12e5faa76ea17c4
[ 2663.706658] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2695.136852] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2739.662589] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2775.528993] Lustre: 15961:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.158@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2788.900089] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2830.368957] LustreError: MGC192.168.206.158@tcp: Connection to MGS (at 192.168.206.158@tcp) was lost; in progress operations using this service will fail
[ 2830.385582] LustreError: Skipped 3 previous similar messages
[ 2840.628671] Lustre: Evicted from MGS (at 192.168.206.158@tcp) after server handle changed from 0xa12e5faa76ea7c64 to 0xa12e5faa76eae533
[ 2840.639166] Lustre: Skipped 3 previous similar messages
[ 2840.660198] Lustre: MGC192.168.206.158@tcp: Connection restored to 192.168.206.158@tcp (at 192.168.206.158@tcp)
[ 2840.673853] Lustre: Skipped 16 previous similar messages
[ 2852.007682] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2881.108089] Lustre: lustre-MDT0000-mdc-ffff94750c65d000: Connection to lustre-MDT0000 (at 192.168.206.158@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2881.146801] Lustre: Skipped 8 previous similar messages
[ 2921.487865] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 3017.404555] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 11:48:45 (1788536925)
[ 3073.349974] Lustre: DEBUG MARKER: oleg658-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3075.470259] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3363.747758] LustreError: lustre-OST0000-osc-ffff94750c65d000: operation ost_write to node 192.168.206.158@tcp failed: rc = -19
[ 3363.756428] LustreError: Skipped 10 previous similar messages
[ 3402.872993] Lustre: DEBUG MARKER: oleg658-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3405.230861] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3694.778123] Lustre: lustre-OST0000-osc-ffff94750c65d000: Connection to lustre-OST0000 (at 192.168.206.158@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3694.806723] Lustre: Skipped 3 previous similar messages
[ 3735.964893] Lustre: DEBUG MARKER: oleg658-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3738.823134] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3991.160318] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 12:04:59 (1788537899)
[ 4007.907035] Lustre: 50628:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788537901/real 1788537901]  req@ffff947505acd500 x1875413862119680/t0(0) o101->lustre-MDT0000-mdc-ffff94750c65d000@192.168.206.158@tcp:12/10 lens 576/1152 e 0 to 1 dl 1788537917 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 4007.953446] Lustre: 50628:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
[ 4008.006530] Lustre: lustre-MDT0000-mdc-ffff94750c65d000: Connection restored to 192.168.206.158@tcp (at 192.168.206.158@tcp)
[ 4008.022805] Lustre: Skipped 6 previous similar messages
[ 4017.213731] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 12:05:24 (1788537924)
[ 4046.713135] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 12:05:53 (1788537953)
[ 4073.127915] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 12:06:21 (1788537981)
[ 4073.601234] Lustre: Mounted lustre-client - version 2.17.58_3_gb2d482c
[ 4105.695196] Lustre: 2372:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788537998/real 1788537998]  req@ffff947505acc700 x1875413862138752/t0(0) o400->lustre-MDT0000-mdc-ffff94750c65d000@192.168.206.158@tcp:12/10 lens 224/224 e 0 to 1 dl 1788538014 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4105.741135] Lustre: 2372:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 4105.752939] LustreError: MGC192.168.206.158@tcp: Connection to MGS (at 192.168.206.158@tcp) was lost; in progress operations using this service will fail
[ 4105.774471] LustreError: Skipped 2 previous similar messages
[ 4115.974209] Lustre: Evicted from MGS (at 192.168.206.158@tcp) after server handle changed from 0xa12e5faa76ec4f3b to 0xa12e5faa76ffb790
[ 4116.002412] Lustre: Skipped 2 previous similar messages
[ 4135.287508] Lustre: DEBUG MARKER: oleg658-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4137.903261] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4140.360503] Lustre: Unmounted lustre-client
[ 4148.860817] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 12:07:36 (1788538056)
[ 4270.559882] Lustre: 2373:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788538163/real 1788538163]  req@ffff947508ac0e00 x1875413862174080/t0(0) o4->lustre-OST0000-osc-ffff94750c65d000@192.168.206.158@tcp:6/4 lens 488/448 e 0 to 1 dl 1788538179 ref 2 fl Rpc:XQr/602/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4270.607146] Lustre: 2373:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 58 previous similar messages
[ 4302.303335] Lustre: lustre-OST0000-osc-ffff94750c65d000: Connection to lustre-OST0000 (at 192.168.206.158@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4302.332171] Lustre: Skipped 13 previous similar messages
[ 4442.406522] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 12:12:30 (1788538350)
[ 4490.501376] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 12:13:18 (1788538398)
[ 4494.053724] Lustre: Unmounted lustre-client
[ 4520.433607] Lustre: Mounted lustre-client - version 2.17.58_3_gb2d482c
[ 4527.909812] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 12:13:55 (1788538435)
[ 4528.175769] LustreError: 56551:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4529.223205] LustreError: 56551:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4529.533546] Lustre: *** cfs_fail_loc=305, val=0***
[ 4552.620082] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 12:14:20 (1788538460)
[ 4553.280584] Lustre: Mounted lustre-client - version 2.17.58_3_gb2d482c
[ 4556.178636] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4566.489847] Lustre: Unmounted lustre-client
[ 4576.447980] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 12:14:43 (1788538483)
[ 4662.737671] LustreError: lustre-MDT0000-mdc-ffff947505733000: operation mds_reint to node 192.168.206.158@tcp failed: rc = -19
[ 4662.745273] LustreError: Skipped 2 previous similar messages
[ 4683.231241] Lustre: 2371:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788538576/real 1788538576]  req@ffff947508aa2a00 x1875413864849280/t0(0) o400->MGC192.168.206.158@tcp@192.168.206.158@tcp:26/25 lens 224/224 e 0 to 1 dl 1788538592 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4683.270550] Lustre: 2371:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 85 previous similar messages
[ 4683.295674] LustreError: MGC192.168.206.158@tcp: Connection to MGS (at 192.168.206.158@tcp) was lost; in progress operations using this service will fail
[ 4693.488176] Lustre: Evicted from MGS (at 192.168.206.158@tcp) after server handle changed from 0xa12e5faa76ffcf14 to 0xa12e5faa770284ae
[ 4693.507968] Lustre: MGC192.168.206.158@tcp: Connection restored to 192.168.206.158@tcp (at 192.168.206.158@tcp)
[ 4693.514831] Lustre: Skipped 23 previous similar messages
[ 4848.971816] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 12:19:16 (1788538756)
[ 4853.992331] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4870.636317] LustreError: MGC192.168.206.158@tcp: Connection to MGS (at 192.168.206.158@tcp) was lost; in progress operations using this service will fail
[ 4870.667823] Lustre: Evicted from MGS (at 192.168.206.158@tcp) after server handle changed from 0xa12e5faa770284ae to 0xa12e5faa77074e72
[ 4870.712115] LustreError: lustre-MDT0000-mdc-ffff947505733000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4884.370428] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 12:19:52 (1788538792)
[ 4884.791197] Lustre: Mounted lustre-client - version 2.17.58_3_gb2d482c
[ 4892.154083] LustreError: lustre-OST0000-osc-ffff947505733000: operation ldlm_enqueue to node 192.168.206.158@tcp failed: rc = -107
[ 4892.175826] LustreError: lustre-OST0000-osc-ffff947505733000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4902.367206] Lustre: lustre-OST0000-osc-ffff947508aa9800: Connection to lustre-OST0000 (at 192.168.206.158@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4902.383861] Lustre: Skipped 13 previous similar messages
[ 4904.765853] Lustre: Unmounted lustre-client
[ 4911.849882] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 12:20:19 (1788538819)
[ 4918.577551] LustreError: lustre-MDT0000-mdc-ffff947505733000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4918.605375] LustreError: 60945:0:(file.c:6233:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4926.012196] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 12:20:33 (1788538833)
[ 4926.151148] LustreError: 61647:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4931.551148] LustreError: 61647:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=0
[ 4931.553534] LustreError: 61647:0:(import.c:755:ptlrpc_connect_import_locked()) already connected
[ 4932.837674] LustreError: lustre-MDT0000-mdc-ffff947505733000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4932.848584] LustreError: 61667:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4932.891470] LustreError: 61672:0:(file.c:6233:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4933.979582] LustreError: 61678:0:(file.c:6233:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4933.992140] LustreError: 61678:0:(file.c:6233:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4936.339670] LustreError: 61701:0:(file.c:6233:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4936.363606] LustreError: 61701:0:(file.c:6233:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4941.036337] LustreError: 61746:0:(file.c:6233:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4941.055302] LustreError: 61746:0:(file.c:6233:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4952.052925] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 12:20:59 (1788538859)
[ 4991.849780] Lustre: DEBUG MARKER: oleg658-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4993.421754] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5004.758092] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 12:21:52 (1788538912)
[ 5043.031458] Lustre: DEBUG MARKER: oleg658-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5044.447728] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5053.172690] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 12:22:41 (1788538961)
[ 5117.026422] Lustre: DEBUG MARKER: oleg658-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5118.828443] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5127.199991] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 12:23:55 (1788539035)
[ 5161.949627] Lustre: DEBUG MARKER: oleg658-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5163.211987] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5168.051045] Lustre: Unmounted lustre-client
[ 5213.057096] Lustre: DEBUG MARKER: oleg658-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5215.798651] Lustre: Mounted lustre-client - version 2.17.58_3_gb2d482c
[ 5221.466406] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 12:25:29 (1788539129)
[ 5223.566171] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5224.863596] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 12:25:33 (1788539133)
[ 5255.217909] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 12:26:03 (1788539163)
[ 5256.391677] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5257.856186] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 12:26:05 (1788539165)
[ 5259.312810] Lustre: *** cfs_fail_loc=805, val=0***
[ 5259.348324] Lustre: Mounted lustre-client - version 2.17.58_3_gb2d482c
[ 5263.540065] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5284.831161] Lustre: 2371:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788539178/real 1788539178]  req@ffff9475081c1500 x1875413866900352/t0(0) o400->lustre-MDT0000-mdc-ffff947504a6c000@192.168.206.158@tcp:12/10 lens 224/224 e 0 to 1 dl 1788539194 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5284.859076] LustreError: MGC192.168.206.158@tcp: Connection to MGS (at 192.168.206.158@tcp) was lost; in progress operations using this service will fail
[ 5284.861048] Lustre: 2371:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5284.920219] Lustre: Evicted from MGS (at 192.168.206.158@tcp) after server handle changed from 0xa12e5faa77075890 to 0xa12e5faa77075c3a
[ 5290.515800] Lustre: Unmounted lustre-client
[ 5296.649707] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 12:26:44 (1788539204)
[ 5322.787108] LustreError: 2369:0:(client.c:3449:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff947507a35c00 x1875413866911232/t90194313221(90194313221) o101->lustre-MDT0000-mdc-ffff947504a6c000@192.168.206.158@tcp:12/10 lens 648/608 e 0 to 0 dl 1788539248 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5322.925049] Lustre: lustre-MDT0000-mdc-ffff947504a6c000: Connection restored to 192.168.206.158@tcp (at 192.168.206.158@tcp)
[ 5322.944655] Lustre: Skipped 16 previous similar messages
[ 5332.205660] Lustre: DEBUG MARKER: oleg658-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5333.599166] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5340.958024] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 12:27:28 (1788539248)
[ 5342.144696] LustreError: lustre-OST0000-osc-ffff947504a6c000: operation ost_write to node 192.168.206.158@tcp failed: rc = -107
[ 5342.160497] LustreError: Skipped 2 previous similar messages
[ 5342.185962] LustreError: lustre-OST0000-osc-ffff947504a6c000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5342.202843] Lustre: 2371:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.206.158@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5342.241243] LustreError: 72846:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff947504a6c000: namespace resource [0x240000400:0x3a62:0x0].0x0 (ffff947506f22700) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5349.489873] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 12:27:37 (1788539257)
[ 5350.739710] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5352.326456] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 12:27:40 (1788539260)
[ 5353.803873] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5355.442335] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 12:27:43 (1788539263)
[ 5356.767503] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5358.160519] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 12:27:46 (1788539266)
[ 5359.366645] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5360.869753] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 12:27:48 (1788539268)
[ 5362.010791] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5363.477294] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 12:27:51 (1788539271)
[ 5365.017983] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5366.712465] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 12:27:54 (1788539274)
[ 5368.279841] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5369.713064] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 12:27:57 (1788539277)
[ 5371.199595] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5372.933654] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 12:28:00 (1788539280)
[ 5374.401984] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5376.023039] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 12:28:03 (1788539283)
[ 5377.432426] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5378.750603] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 12:28:07 (1788539287)
[ 5380.278482] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5381.862377] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 12:28:09 (1788539289)
[ 5384.319906] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5386.276665] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 12:28:13 (1788539293)
[ 5408.253522] Lustre: 68308:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.158@tcp: IR log lustre-cliir failed, not fatal: rc = -5
[ 5416.053876] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 12:28:44 (1788539324)
[ 5444.625618] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 12:29:12 (1788539352)
[ 5445.311798] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5455.564488] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 12:29:23 (1788539363)
[ 5456.063280] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5466.802298] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 12:29:34 (1788539374)
[ 5467.114736] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5474.869854] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 12:29:42 (1788539382)
[ 5475.135559] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5482.463378] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 12:29:50 (1788539390)
[ 5482.895273] Lustre: *** cfs_fail_loc=510, val=3***
[ 5490.666939] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 12:29:58 (1788539398)
[ 5491.141609] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5501.561922] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 12:30:09 (1788539409)
[ 5502.230568] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5561.311406] Lustre: lustre-OST0001-osc-ffff947504a6c000: Connection to lustre-OST0001 (at 192.168.206.158@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5561.348481] Lustre: Skipped 15 previous similar messages
[ 5568.880531] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 12:31:16 (1788539476)
[ 5569.205187] LustreError: 83051:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5572.165877] LustreError: lustre-MDT0000-mdc-ffff947504a6c000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5572.197392] LustreError: 83066:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff947504a6c000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff947507056800) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5573.280830] LustreError: 83051:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5575.369592] LustreError: 83082:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5578.119331] LustreError: lustre-MDT0000-mdc-ffff947504a6c000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5578.132865] LustreError: 83096:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff947504a6c000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff947507b1c500) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5578.152950] LustreError: 83096:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5579.439401] LustreError: 83082:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5586.239987] LustreError: lustre-MDT0000-mdc-ffff947504a6c000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5586.638749] LustreError: 83138:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5589.312427] LustreError: lustre-MDT0000-mdc-ffff947504a6c000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5589.361653] LustreError: 83157:0:(file.c:6233:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5589.373840] LustreError: 83157:0:(file.c:6233:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5590.444539] LustreError: 83163:0:(file.c:6233:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5590.458422] LustreError: 83163:0:(file.c:6233:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5590.720805] LustreError: 83138:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5590.725098] LustreError: 83138:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff947504a6c000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5592.732805] LustreError: 83186:0:(file.c:6233:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5592.739650] LustreError: 83186:0:(file.c:6233:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5596.223620] LustreError: 83213:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5599.035904] LustreError: lustre-MDT0000-mdc-ffff947504a6c000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5599.055681] LustreError: 83227:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff947504a6c000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff947507056200) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5599.093948] LustreError: 83227:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5600.311165] LustreError: 83213:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5607.247391] LustreError: lustre-MDT0000-mdc-ffff947504a6c000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5607.556309] LustreError: 83269:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5610.402378] LustreError: lustre-MDT0000-mdc-ffff947504a6c000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5610.418691] LustreError: 83283:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff947504a6c000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff947507056000) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5610.455545] LustreError: 83283:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5611.623181] LustreError: 83269:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5614.801292] LustreError: lustre-MDT0000-mdc-ffff947504a6c000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5614.913521] LustreError: 83316:0:(file.c:6233:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5614.925545] LustreError: 83316:0:(file.c:6233:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5615.791951] LustreError: 83297:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff947504a6c000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5624.112536] LustreError: 83368:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5624.124953] LustreError: 83368:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5627.695617] LustreError: lustre-MDT0000-mdc-ffff947504a6c000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5627.759453] LustreError: 83391:0:(file.c:6233:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5627.770360] LustreError: 83391:0:(file.c:6233:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5628.209610] LustreError: 83368:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5628.224586] LustreError: 83368:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5628.232667] LustreError: 83368:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff947504a6c000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5638.883694] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 12:32:26 (1788539546)
[ 5658.090565] LustreError: 68306:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5658.096374] LustreError: 68306:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5662.095125] LustreError: 68306:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5662.106582] LustreError: 68306:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5668.009594] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 12:32:56 (1788539576)
[ 5717.516924] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 12:33:45 (1788539625)
[ 5781.599980] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 12:34:49 (1788539689)
[ 5815.218176] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 12:35:22 (1788539722)
[ 5815.731104] Lustre: Mounted lustre-client - version 2.17.58_3_gb2d482c
[ 5939.624604] Lustre: Unmounted lustre-client
[ 5947.274780] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 12:37:35 (1788539855)
[ 5947.731442] LustreError: 87696:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5950.443131] LustreError: lustre-OST0000-osc-ffff947504a6c000: operation ldlm_enqueue to node 192.168.206.158@tcp failed: rc = -107
[ 5950.465777] LustreError: Skipped 9 previous similar messages
[ 5950.491436] LustreError: lustre-OST0000-osc-ffff947504a6c000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5950.510117] LustreError: 87778:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff947504a6c000: namespace resource [0x240000400:0x3a8d:0x0].0x0 (ffff947506f22000) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5950.525992] LustreError: 87778:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5950.879125] LustreError: 87696:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5950.883487] Lustre: 2373:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.206.158@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5958.600282] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 12:37:45 (1788539865)
[ 6015.967180] Lustre: 88364:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788539870/real 1788539870]  req@ffff947507033100 x1875413867081600/t0(0) o101->lustre-MDT0000-mdc-ffff947504a6c000@192.168.206.158@tcp:12/10 lens 328/344 e 0 to 1 dl 1788539925 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 6015.997958] Lustre: 88364:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 16 previous similar messages
[ 6016.030221] Lustre: lustre-MDT0000-mdc-ffff947504a6c000: Connection restored to 192.168.206.158@tcp (at 192.168.206.158@tcp)
[ 6016.041113] Lustre: Skipped 16 previous similar messages
[ 6022.723248] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 12:38:50 (1788539930)
[ 6023.844453] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 6025.413171] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 12:38:53 (1788539933)
[ 6088.813354] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6090.683915] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 12:39:58 (1788539998)
[ 6149.069184] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 12:40:56 (1788540056)
[ 6151.900160] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6153.767896] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 12:41:01 (1788540061)
[ 6155.382879] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6157.742903] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 12:41:05 (1788540065)
[ 6190.073685] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 12:41:38 (1788540098)
[ 6200.528545] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6228.767203] LustreError: MGC192.168.206.158@tcp: Connection to MGS (at 192.168.206.158@tcp) was lost; in progress operations using this service will fail
[ 6228.777454] LustreError: Skipped 3 previous similar messages
[ 6228.791622] Lustre: Evicted from MGS (at 192.168.206.158@tcp) after server handle changed from 0xa12e5faa770766f9 to 0xa12e5faa77077ffe
[ 6228.805257] Lustre: Skipped 3 previous similar messages
[ 6228.808634] Lustre: lustre-MDT0000-mdc-ffff947504a6c000: Connection to lustre-MDT0000 (at 192.168.206.158@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6228.819661] Lustre: Skipped 16 previous similar messages
[ 6240.745713] Lustre: DEBUG MARKER: oleg658-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6242.245983] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6250.982898] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 12:42:38 (1788540158)
[ 6252.777325] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6254.216984] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 12:42:42 (1788540162)
[ 6269.940924] LustreError: MGC192.168.206.158@tcp: Connection to MGS (at 192.168.206.158@tcp) was lost; in progress operations using this service will fail
[ 6269.973920] Lustre: Evicted from MGS (at 192.168.206.158@tcp) after server handle changed from 0xa12e5faa77077ffe to 0xa12e5faa770783b6
[ 6278.530368] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 12:43:06 (1788540186)
[ 6327.066429] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 12:43:54 (1788540234)
[ 6393.642223] Lustre: DEBUG MARKER: oleg658-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6399.122947] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6483.871502] LustreError: MGC192.168.206.158@tcp: Connection to MGS (at 192.168.206.158@tcp) was lost; in progress operations using this service will fail
[ 6483.881247] LustreError: Skipped 1 previous similar message
[ 6483.897201] Lustre: Evicted from MGS (at 192.168.206.158@tcp) after server handle changed from 0xa12e5faa77078752 to 0xa12e5faa77079505
[ 6483.905706] Lustre: Skipped 1 previous similar message
[ 6496.138624] Lustre: DEBUG MARKER: oleg658-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6497.565778] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6532.182982] Lustre: DEBUG MARKER: oleg658-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6533.718423] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6663.073503] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 12:49:30 (1788540570)
[ 6672.995275] LustreError: lustre-OST0000-osc-ffff947504a6c000: operation ost_setattr to node 192.168.206.158@tcp failed: rc = -19
[ 6673.005057] LustreError: Skipped 326 previous similar messages
[ 6720.864217] Lustre: DEBUG MARKER: oleg658-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6723.668775] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6972.948983] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 12:54:40 (1788540880)
[ 7134.182063] Lustre: lustre-MDT0000-mdc-ffff947504a6c000: Connection to lustre-MDT0000 (at 192.168.206.158@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7134.202211] Lustre: Skipped 6 previous similar messages
[ 7149.548128] LustreError: MGC192.168.206.158@tcp: Connection to MGS (at 192.168.206.158@tcp) was lost; in progress operations using this service will fail
[ 7149.560306] LustreError: Skipped 1 previous similar message
[ 7149.590384] Lustre: Evicted from MGS (at 192.168.206.158@tcp) after server handle changed from 0xa12e5faa77079918 to 0xa12e5faa77172077
[ 7149.609320] Lustre: Skipped 1 previous similar message
[ 7149.621240] Lustre: MGC192.168.206.158@tcp: Connection restored to 192.168.206.158@tcp (at 192.168.206.158@tcp)
[ 7149.635315] Lustre: Skipped 12 previous similar messages
[ 7184.162940] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 12:58:11 (1788541091)
[ 7185.898495] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7188.029716] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 12:58:15 (1788541095)
[ 7190.252198] LustreError: lustre-MDT0000-mdc-ffff947504a6c000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7198.520400] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 12:58:26 (1788541106)
[ 7366.378157] LustreError: lustre-OST0000-osc-ffff947504a6c000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7368.744146] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 13:01:16 (1788541276)
[ 7393.119137] Lustre: 2372:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788541282/real 1788541282]  req@ffff947509b56d80 x1875413884720896/t0(0) o4->lustre-OST0000-osc-ffff947504a6c000@192.168.206.158@tcp:6/4 lens 4584/448 e 0 to 1 dl 1788541302 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7393.147549] Lustre: 2372:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 22 previous similar messages
[ 7411.714884] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 13:01:59 (1788541319)
[ 7413.525885] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7415.785097] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 13:02:03 (1788541323)
[ 7417.773277] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7420.050886] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 13:02:07 (1788541327)
[ 7456.534617] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 13:02:43 (1788541363)
[ 7499.751209] LustreError: MGC192.168.206.158@tcp: Connection to MGS (at 192.168.206.158@tcp) was lost; in progress operations using this service will fail
[ 7499.775367] Lustre: Evicted from MGS (at 192.168.206.158@tcp) after server handle changed from 0xa12e5faa77172077 to 0xa12e5faa77174ba4
[ 7499.782740] Lustre: Skipped 1 previous similar message
[ 7517.861612] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 13:03:45 (1788541425)
[ 7519.738573] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7521.762560] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 13:03:49 (1788541429)
[ 7523.354256] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7525.073539] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 13:03:52 (1788541432)
[ 7525.344086] Lustre: 2371:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788541398/real 1788541398]  req@ffff94753095a680 x1875413884877440/t0(0) o400->lustre-MDT0000-mdc-ffff947504a6c000@192.168.206.158@tcp:12/10 lens 224/224 e 0 to 1 dl 1788541434 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7536.421865] Lustre: Unmounted lustre-client
[ 7538.311606] Lustre: Mounted lustre-client - version 2.17.58_3_gb2d482c
[ 7540.995444] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7581.202640] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2