[ 2020.305181] LustreError: lustre-OST0000-osc-ffff923f84713800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2020.324969] Lustre: 2364:0:(llite_lib.c:4341:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.140@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 2027.333667] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 08:26:30 (1788524790)
[ 2029.355879] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 2031.264503] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 08:26:34 (1788524794)
[ 2033.277418] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 2035.097443] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 08:26:38 (1788524798)
[ 2057.186722] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 192.168.202.140@tcp) was lost; in progress operations using this service will fail
[ 2067.381319] Lustre: Evicted from MGS (at 192.168.202.140@tcp) after server handle changed from 0x676b594aef559d88 to 0x676b594aef55b591
[ 2171.906327] LustreError: lustre-MDT0000-mdc-ffff923f84713800: operation ldlm_enqueue to node 192.168.202.140@tcp failed: rc = -19
[ 2171.924237] LustreError: Skipped 3 previous similar messages
[ 2190.303330] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 192.168.202.140@tcp) was lost; in progress operations using this service will fail
[ 2200.495561] Lustre: Evicted from MGS (at 192.168.202.140@tcp) after server handle changed from 0x676b594aef55b591 to 0x676b594aef578b71
[ 2216.986424] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 08:29:39 (1788524979)
[ 2217.554195] Lustre: *** cfs_fail_loc=305, val=0***
[ 2217.557867] Lustre: Skipped 2 previous similar messages
[ 2259.935415] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 192.168.202.140@tcp) was lost; in progress operations using this service will fail
[ 2259.960914] Lustre: Evicted from MGS (at 192.168.202.140@tcp) after server handle changed from 0x676b594aef578b71 to 0x676b594aef579986
[ 2259.968536] Lustre: MGC192.168.202.140@tcp: Connection restored to 192.168.202.140@tcp (at 192.168.202.140@tcp)
[ 2259.974883] Lustre: Skipped 13 previous similar messages
[ 2261.423832] Lustre: 15966:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.140@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2269.301889] Lustre: lustre-MDT0000-mdc-ffff923f84713800: Connection to lustre-MDT0000 (at 192.168.202.140@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2269.317666] Lustre: Skipped 11 previous similar messages
[ 2284.791584] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2286.903834] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2296.976168] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 08:30:59 (1788525059)
[ 2311.147594] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 192.168.202.140@tcp) was lost; in progress operations using this service will fail
[ 2311.203846] Lustre: Evicted from MGS (at 192.168.202.140@tcp) after server handle changed from 0x676b594aef579986 to 0x676b594aef579ecd
[ 2311.299786] LustreError: lustre-MDT0000-mdc-ffff923f84713800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2337.404904] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 08:31:39 (1788525099)
[ 2353.158724] LustreError: lustre-OST0000-osc-ffff923f84713800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2373.331702] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 08:32:16 (1788525136)
[ 2386.844835] LustreError: lustre-MDT0000-mdc-ffff923f84713800: operation mds_reint to node 192.168.202.140@tcp failed: rc = -19
[ 2404.321838] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 192.168.202.140@tcp) was lost; in progress operations using this service will fail
[ 2414.589791] Lustre: Evicted from MGS (at 192.168.202.140@tcp) after server handle changed from 0x676b594aef579ecd to 0x676b594aef57ecad
[ 2432.453149] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2435.443406] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2521.119243] Lustre: 2364:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788525269/real 1788525269]  req@ffff923f86334000 x1875402193135616/t0(0) o400->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 224/224 e 0 to 1 dl 1788525285 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2521.137740] Lustre: 2364:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 15 previous similar messages
[ 2521.143450] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 192.168.202.140@tcp) was lost; in progress operations using this service will fail
[ 2531.364519] Lustre: Evicted from MGS (at 192.168.202.140@tcp) after server handle changed from 0x676b594aef57ecad to 0x676b594aef5985b6
[ 2552.879214] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2555.690437] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2622.970830] LustreError: lustre-MDT0000-mdc-ffff923f84713800: operation ldlm_enqueue to node 192.168.202.140@tcp failed: rc = -19
[ 2622.985915] LustreError: Skipped 3 previous similar messages
[ 2642.722247] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 192.168.202.140@tcp) was lost; in progress operations using this service will fail
[ 2652.919031] Lustre: Evicted from MGS (at 192.168.202.140@tcp) after server handle changed from 0x676b594aef5985b6 to 0x676b594aef5b1cf1
[ 2670.371447] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2672.728575] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2703.712146] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 08:37:46 (1788525466)
[ 2729.504289] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 192.168.202.140@tcp) was lost; in progress operations using this service will fail
[ 2738.736881] Lustre: Evicted from MGS (at 192.168.202.140@tcp) after server handle changed from 0x676b594aef5b1cf1 to 0x676b594aef5bd004
[ 2749.772878] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2797.206464] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2834.578873] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2869.727441] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 192.168.202.140@tcp) was lost; in progress operations using this service will fail
[ 2869.736629] LustreError: Skipped 2 previous similar messages
[ 2879.929026] Lustre: Evicted from MGS (at 192.168.202.140@tcp) after server handle changed from 0x676b594aef5c2f80 to 0x676b594aef5c4911
[ 2879.943329] Lustre: Skipped 2 previous similar messages
[ 2879.957831] Lustre: MGC192.168.202.140@tcp: Connection restored to 192.168.202.140@tcp (at 192.168.202.140@tcp)
[ 2879.978321] Lustre: Skipped 16 previous similar messages
[ 2888.308202] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2912.188602] LustreError: lustre-MDT0000-mdc-ffff923f84713800: operation mds_close to node 192.168.202.140@tcp failed: rc = -19
[ 2912.199918] LustreError: Skipped 4 previous similar messages
[ 2912.207769] Lustre: lustre-MDT0000-mdc-ffff923f84713800: Connection to lustre-MDT0000 (at 192.168.202.140@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2912.218200] Lustre: Skipped 9 previous similar messages
[ 2949.471174] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 3010.296179] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 3108.013365] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 08:44:31 (1788525871)
[ 3166.561031] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3168.883653] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3454.983631] LustreError: lustre-OST0000-osc-ffff923f84713800: operation ldlm_enqueue to node 192.168.202.140@tcp failed: rc = -107
[ 3454.997089] LustreError: Skipped 7 previous similar messages
[ 3504.373671] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3507.373076] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3787.883251] Lustre: lustre-OST0000-osc-ffff923f84713800: Connection to lustre-OST0000 (at 192.168.202.140@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3787.904634] Lustre: Skipped 4 previous similar messages
[ 3832.765408] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3835.729231] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4085.026539] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 09:00:47 (1788526847)
[ 4103.647165] Lustre: 50605:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788526851/real 1788526851]  req@ffff923f847a0e00 x1875402210878464/t0(0) o101->lustre-MDT0000-mdc-ffff923f84713800@192.168.202.140@tcp:12/10 lens 576/1152 e 0 to 1 dl 1788526867 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 4103.678305] Lustre: 50605:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
[ 4103.711242] Lustre: lustre-MDT0000-mdc-ffff923f84713800: Connection restored to 192.168.202.140@tcp (at 192.168.202.140@tcp)
[ 4103.723050] Lustre: Skipped 8 previous similar messages
[ 4113.420375] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 09:01:16 (1788526876)
[ 4140.084382] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 09:01:42 (1788526902)
[ 4167.462364] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 09:02:10 (1788526930)
[ 4168.392773] Lustre: Mounted lustre-client - version 2.17.58_2_gdacfdd2
[ 4200.415625] Lustre: 2362:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788526948/real 1788526948]  req@ffff923f850e3800 x1875402210897280/t0(0) o400->lustre-MDT0000-mdc-ffff923f84713800@192.168.202.140@tcp:12/10 lens 224/224 e 0 to 1 dl 1788526964 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4200.448495] Lustre: 2362:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 4200.476047] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 192.168.202.140@tcp) was lost; in progress operations using this service will fail
[ 4200.487967] LustreError: Skipped 3 previous similar messages
[ 4210.684747] Lustre: Evicted from MGS (at 192.168.202.140@tcp) after server handle changed from 0x676b594aef5dd6eb to 0x676b594aef6e560d
[ 4210.696900] Lustre: Skipped 3 previous similar messages
[ 4210.729691] Lustre: MGC192.168.202.140@tcp: Connection restored to 192.168.202.140@tcp (at 192.168.202.140@tcp)
[ 4210.736411] Lustre: Skipped 2 previous similar messages
[ 4225.803095] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4227.650915] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4230.233540] Lustre: Unmounted lustre-client
[ 4237.233723] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 09:03:19 (1788526999)
[ 4359.711166] Lustre: 2365:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788527108/real 1788527108]  req@ffff923f857c7100 x1875402210932096/t0(0) o4->lustre-OST0000-osc-ffff923f84713800@192.168.202.140@tcp:6/4 lens 488/448 e 0 to 1 dl 1788527124 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4359.732538] Lustre: 2365:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 70 previous similar messages
[ 4376.053130] Lustre: lustre-OST0000-osc-ffff923f84713800: Connection restored to 192.168.202.140@tcp (at 192.168.202.140@tcp)
[ 4376.067653] Lustre: Skipped 8 previous similar messages
[ 4392.355916] Lustre: lustre-OST0000-osc-ffff923f84713800: Connection to lustre-OST0000 (at 192.168.202.140@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4392.366583] Lustre: Skipped 13 previous similar messages
[ 4536.403977] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 09:08:18 (1788527298)
[ 4585.794456] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 09:09:08 (1788527348)
[ 4589.786683] Lustre: Unmounted lustre-client
[ 4615.078176] Lustre: Mounted lustre-client - version 2.17.58_2_gdacfdd2
[ 4622.495112] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 09:09:45 (1788527385)
[ 4622.926429] LustreError: 56542:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4623.986649] LustreError: 56542:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4624.140550] Lustre: *** cfs_fail_loc=305, val=0***
[ 4648.317340] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 09:10:10 (1788527410)
[ 4648.769631] Lustre: Mounted lustre-client - version 2.17.58_2_gdacfdd2
[ 4651.498516] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4661.829730] Lustre: Unmounted lustre-client
[ 4670.367693] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 09:10:33 (1788527433)
[ 4766.994492] LustreError: lustre-MDT0000-mdc-ffff923f8565c000: operation mds_reint to node 192.168.202.140@tcp failed: rc = -19
[ 4767.008033] LustreError: Skipped 1 previous similar message
[ 4788.193821] Lustre: 2365:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788527536/real 1788527536]  req@ffff923f88c91500 x1875402213601408/t0(0) o400->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 224/224 e 0 to 1 dl 1788527552 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4788.229509] Lustre: 2365:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 79 previous similar messages
[ 4788.241940] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 192.168.202.140@tcp) was lost; in progress operations using this service will fail
[ 4798.449596] Lustre: Evicted from MGS (at 192.168.202.140@tcp) after server handle changed from 0x676b594aef6e6d2f to 0x676b594aef711ecb
[ 4798.460170] Lustre: MGC192.168.202.140@tcp: Connection restored to 192.168.202.140@tcp (at 192.168.202.140@tcp)
[ 4798.472554] Lustre: Skipped 11 previous similar messages
[ 4968.950167] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 09:15:31 (1788527731)
[ 4976.410483] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4991.984417] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 192.168.202.140@tcp) was lost; in progress operations using this service will fail
[ 4992.034838] Lustre: Evicted from MGS (at 192.168.202.140@tcp) after server handle changed from 0x676b594aef711ecb to 0x676b594aef75f029
[ 4992.167264] LustreError: lustre-MDT0000-mdc-ffff923f8565c000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5010.614295] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 09:16:12 (1788527772)
[ 5011.214493] Lustre: Mounted lustre-client - version 2.17.58_2_gdacfdd2
[ 5019.056838] LustreError: lustre-OST0000-osc-ffff923f8565c000: operation ldlm_enqueue to node 192.168.202.140@tcp failed: rc = -107
[ 5019.072465] Lustre: lustre-OST0000-osc-ffff923f8565c000: Connection to lustre-OST0000 (at 192.168.202.140@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5019.091788] Lustre: Skipped 12 previous similar messages
[ 5019.110886] LustreError: lustre-OST0000-osc-ffff923f8565c000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5019.131952] LustreError: 60265:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff923f8565c000: namespace resource [0x240000400:0x3523:0x0].0x0 (ffff923f88c1aa00) refcount nonzero (2) after lock cleanup; forcing cleanup.
[ 5031.846618] Lustre: Unmounted lustre-client
[ 5040.349420] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 09:16:42 (1788527802)
[ 5048.211279] LustreError: lustre-MDT0000-mdc-ffff923f8565c000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5048.226768] LustreError: 60937:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 5057.570099] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 09:17:00 (1788527820)
[ 5057.855834] LustreError: 61641:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 5062.103992] LustreError: lustre-MDT0000-mdc-ffff923f8565c000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5062.129184] LustreError: 61658:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 5062.148497] LustreError: 61641:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=718
[ 5062.162643] LustreError: 61641:0:(import.c:761:ptlrpc_connect_import_locked()) already connecting
[ 5062.224606] LustreError: 61663:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5063.332683] LustreError: 61670:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5063.345460] LustreError: 61670:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5065.928665] LustreError: 61692:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5065.949851] LustreError: 61692:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5069.937884] LustreError: 61733:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5069.947597] LustreError: 61733:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 23 previous similar messages
[ 5082.126134] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 09:17:24 (1788527844)
[ 5124.222447] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5125.791353] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5139.183776] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 09:18:21 (1788527901)
[ 5185.410672] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5187.196728] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5201.246928] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 09:19:23 (1788527963)
[ 5271.856551] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5273.627922] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5282.785944] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 09:20:45 (1788528045)
[ 5327.607618] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5329.748247] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5336.598278] Lustre: Unmounted lustre-client
[ 5398.202038] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5401.199942] Lustre: Mounted lustre-client - version 2.17.58_2_gdacfdd2
[ 5408.967643] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 09:22:51 (1788528171)
[ 5411.931692] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5414.092859] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 09:22:56 (1788528176)
[ 5448.608605] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 09:23:30 (1788528210)
[ 5450.532800] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5452.917648] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 09:23:35 (1788528215)
[ 5454.885636] Lustre: *** cfs_fail_loc=805, val=0***
[ 5454.949305] Lustre: Mounted lustre-client - version 2.17.58_2_gdacfdd2
[ 5463.368569] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5487.071455] Lustre: 2364:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788528235/real 1788528235]  req@ffff923f87b30700 x1875402215668352/t0(0) o400->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 224/224 e 0 to 1 dl 1788528251 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5487.100673] Lustre: 2364:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5487.114482] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 192.168.202.140@tcp) was lost; in progress operations using this service will fail
[ 5487.144897] Lustre: Evicted from MGS (at 192.168.202.140@tcp) after server handle changed from 0x676b594aef75fa55 to 0x676b594aef75fc70
[ 5487.171824] Lustre: MGC192.168.202.140@tcp: Connection restored to 192.168.202.140@tcp (at 192.168.202.140@tcp)
[ 5487.179088] Lustre: Skipped 11 previous similar messages
[ 5488.319964] Lustre: 68313:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.140@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 5506.440234] Lustre: Unmounted lustre-client
[ 5519.723379] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 09:24:42 (1788528282)
[ 5562.432319] LustreError: 2361:0:(client.c:3449:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff923f87bfea00 x1875402215680768/t94489280516(94489280516) o101->lustre-MDT0000-mdc-ffff923f83c16800@192.168.202.140@tcp:12/10 lens 664/608 e 0 to 0 dl 1788528342 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5575.495409] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5578.151547] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5589.316715] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 09:25:52 (1788528352)
[ 5591.056369] LustreError: lustre-OST0000-osc-ffff923f83c16800: operation ost_write to node 192.168.202.140@tcp failed: rc = -107
[ 5591.073457] LustreError: Skipped 2 previous similar messages
[ 5591.101898] LustreError: lustre-OST0000-osc-ffff923f83c16800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5591.126770] Lustre: 2365:0:(llite_lib.c:4341:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.140@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5591.183526] LustreError: 72858:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff923f83c16800: namespace resource [0x240000400:0x3582:0x0].0x0 (ffff923f88d6f500) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5600.033059] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 09:26:03 (1788528363)
[ 5601.507491] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5603.157929] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 09:26:06 (1788528366)
[ 5605.056935] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5607.590714] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 09:26:10 (1788528370)
[ 5609.752622] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5611.803596] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 09:26:14 (1788528374)
[ 5613.853382] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5616.078335] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 09:26:18 (1788528378)
[ 5618.268770] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5620.699235] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 09:26:22 (1788528382)
[ 5622.663779] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5624.504439] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 09:26:27 (1788528387)
[ 5626.168474] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5628.461809] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 09:26:30 (1788528390)
[ 5630.562270] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5632.783526] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 09:26:35 (1788528395)
[ 5634.704246] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5636.558174] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 09:26:39 (1788528399)
[ 5638.611160] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5640.776274] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 09:26:43 (1788528403)
[ 5642.754181] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5644.835960] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 09:26:47 (1788528407)
[ 5647.732445] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5650.272387] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 09:26:52 (1788528412)
[ 5669.857793] Lustre: lustre-MDT0000-mdc-ffff923f83c16800: Connection to lustre-MDT0000 (at 192.168.202.140@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5669.872782] Lustre: Skipped 13 previous similar messages
[ 5685.690392] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 09:27:28 (1788528448)
[ 5716.940442] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 09:27:59 (1788528479)
[ 5717.399045] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5728.798963] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 09:28:11 (1788528491)
[ 5729.382408] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5741.338521] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 09:28:23 (1788528503)
[ 5741.762471] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5752.450888] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 09:28:34 (1788528514)
[ 5752.927651] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5762.275425] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 09:28:45 (1788528525)
[ 5762.633610] Lustre: *** cfs_fail_loc=510, val=3***
[ 5771.439983] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 09:28:54 (1788528534)
[ 5771.812734] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5784.624037] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 09:29:06 (1788528546)
[ 5785.003244] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5851.843183] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 09:30:14 (1788528614)
[ 5852.323489] LustreError: 83066:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5855.256296] LustreError: lustre-MDT0000-mdc-ffff923f83c16800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5855.268302] LustreError: 83080:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff923f83c16800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff923f88c1ae00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5856.399144] LustreError: 83066:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5858.552936] LustreError: 83096:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5861.590692] LustreError: lustre-MDT0000-mdc-ffff923f83c16800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5861.619770] LustreError: 83110:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff923f83c16800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff923f88c1ae00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5861.641955] LustreError: 83110:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5862.623161] LustreError: 83096:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5869.543372] LustreError: lustre-MDT0000-mdc-ffff923f83c16800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5869.905910] LustreError: 83151:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5873.132861] LustreError: lustre-MDT0000-mdc-ffff923f83c16800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5873.235132] LustreError: 83171:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5873.245623] LustreError: 83171:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5873.999379] LustreError: 83151:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5874.054402] LustreError: 83151:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff923f83c16800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5874.411487] LustreError: 83177:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5874.426789] LustreError: 83177:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5876.846146] LustreError: 83199:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5876.860558] LustreError: 83199:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5880.554598] LustreError: 83226:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5883.654556] LustreError: lustre-MDT0000-mdc-ffff923f83c16800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5883.679252] LustreError: 83240:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff923f83c16800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff923f88c1a900) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5883.711229] LustreError: 83240:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5884.655202] LustreError: 83226:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5891.755960] LustreError: lustre-MDT0000-mdc-ffff923f83c16800: operation ldlm_enqueue to node 192.168.202.140@tcp failed: rc = -107
[ 5891.767841] LustreError: Skipped 5 previous similar messages
[ 5891.787556] LustreError: lustre-MDT0000-mdc-ffff923f83c16800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5892.217293] LustreError: 83282:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5895.364216] LustreError: lustre-MDT0000-mdc-ffff923f83c16800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5895.392556] LustreError: 83296:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff923f83c16800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff923f88d6f200) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5895.422147] LustreError: 83296:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5895.448854] LustreError: 83301:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5895.460793] LustreError: 83301:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5896.287169] LustreError: 83282:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5899.182381] LustreError: lustre-MDT0000-mdc-ffff923f83c16800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5900.478455] LustreError: 83309:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff923f83c16800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5908.526086] LustreError: 83380:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5908.551129] LustreError: 83380:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5911.948434] LustreError: lustre-MDT0000-mdc-ffff923f83c16800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5912.012656] LustreError: 83402:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5912.025571] LustreError: 83402:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 5912.527217] LustreError: 83380:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5912.536764] LustreError: 83380:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5912.554698] LustreError: 83380:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff923f83c16800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5924.790933] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 09:31:27 (1788528687)
[ 5944.766603] LustreError: 68310:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5944.777472] LustreError: 68310:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5948.848171] LustreError: 68310:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5948.853885] LustreError: 68310:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5956.000429] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 09:31:58 (1788528718)
[ 6007.909627] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 09:32:50 (1788528770)
[ 6074.126645] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 09:33:56 (1788528836)
[ 6109.152426] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 09:34:31 (1788528871)
[ 6109.582176] Lustre: Mounted lustre-client - version 2.17.58_2_gdacfdd2
[ 6233.626664] Lustre: Unmounted lustre-client
[ 6240.530767] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 09:36:43 (1788529003)
[ 6241.175101] LustreError: 87708:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 6243.270155] LustreError: lustre-OST0000-osc-ffff923f83c16800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6243.288585] LustreError: 87790:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff923f83c16800: namespace resource [0x240000400:0x35ad:0x0].0x0 (ffff923f88c1a200) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6243.304410] LustreError: 87790:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 6243.599749] LustreError: 87708:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout interrupted
[ 6243.609647] Lustre: 2362:0:(llite_lib.c:4341:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.140@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 6249.135916] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 09:36:52 (1788529012)
[ 6305.759186] Lustre: 88374:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788529015/real 1788529015]  req@ffff923f8857f480 x1875402215855616/t0(0) o101->lustre-MDT0000-mdc-ffff923f83c16800@192.168.202.140@tcp:12/10 lens 328/344 e 0 to 1 dl 1788529070 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 6305.807736] Lustre: 88374:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 20 previous similar messages
[ 6305.821441] Lustre: lustre-MDT0000-mdc-ffff923f83c16800: Connection to lustre-MDT0000 (at 192.168.202.140@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6305.832230] Lustre: Skipped 14 previous similar messages
[ 6305.853905] Lustre: lustre-MDT0000-mdc-ffff923f83c16800: Connection restored to 192.168.202.140@tcp (at 192.168.202.140@tcp)
[ 6305.863220] Lustre: Skipped 18 previous similar messages
[ 6313.984521] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 09:37:57 (1788529077)
[ 6315.431560] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 6317.141666] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 09:38:00 (1788529080)
[ 6381.415240] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6383.066538] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 09:39:06 (1788529146)
[ 6442.373717] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 09:40:04 (1788529204)
[ 6445.210288] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6447.347755] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 09:40:09 (1788529209)
[ 6449.231940] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6451.353672] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 09:40:13 (1788529213)
[ 6493.790886] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 09:40:56 (1788529256)
[ 6506.957152] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6538.210657] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 192.168.202.140@tcp) was lost; in progress operations using this service will fail
[ 6538.214964] LustreError: Skipped 2 previous similar messages
[ 6538.233800] Lustre: Evicted from MGS (at 192.168.202.140@tcp) after server handle changed from 0x676b594aef7608a2 to 0x676b594aef7620dc
[ 6538.247585] Lustre: Skipped 2 previous similar messages
[ 6540.208599] Lustre: 68313:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.140@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6561.700653] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6563.284820] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6574.481892] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 09:42:16 (1788529336)
[ 6577.818231] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6580.256372] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 09:42:22 (1788529342)
[ 6595.564461] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 192.168.202.140@tcp) was lost; in progress operations using this service will fail
[ 6595.583871] Lustre: Evicted from MGS (at 192.168.202.140@tcp) after server handle changed from 0x676b594aef7620dc to 0x676b594aef762597
[ 6614.849937] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 09:42:57 (1788529377)
[ 6668.140769] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 09:43:51 (1788529431)
[ 6675.011885] LustreError: lustre-OST0000-osc-ffff923f83c16800: operation ost_setattr to node 192.168.202.140@tcp failed: rc = -107
[ 6675.023637] LustreError: Skipped 4 previous similar messages
[ 6731.846034] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6736.397052] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6824.928669] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 192.168.202.140@tcp) was lost; in progress operations using this service will fail
[ 6824.943855] LustreError: Skipped 1 previous similar message
[ 6835.187422] Lustre: Evicted from MGS (at 192.168.202.140@tcp) after server handle changed from 0x676b594aef76280d to 0x676b594aef7635f1
[ 6835.196032] Lustre: Skipped 1 previous similar message
[ 6854.616357] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6856.403689] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6905.847638] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6907.685245] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6925.281698] Lustre: 2364:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788529629/real 1788529629]  req@ffff923f9f709880 x1875402222385920/t0(0) o400->lustre-MDT0000-mdc-ffff923f83c16800@192.168.202.140@tcp:12/10 lens 224/224 e 0 to 1 dl 1788529689 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6925.334608] Lustre: 2364:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 20 previous similar messages
[ 7087.080749] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 09:50:49 (1788529849)
[ 7099.720936] Lustre: lustre-OST0000-osc-ffff923f83c16800: Connection to lustre-OST0000 (at 192.168.202.140@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7099.747065] Lustre: Skipped 8 previous similar messages
[ 7165.251923] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 7171.966702] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 7483.564292] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 09:57:26 (1788530246)
[ 7669.728458] Lustre: 2363:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788530418/real 1788530418]  req@ffff923f8acff800 x1875402233550592/t0(0) o400->lustre-MDT0000-mdc-ffff923f83c16800@192.168.202.140@tcp:12/10 lens 224/224 e 0 to 1 dl 1788530434 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7669.747470] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 192.168.202.140@tcp) was lost; in progress operations using this service will fail
[ 7669.781664] Lustre: 2363:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
[ 7669.815566] LustreError: Skipped 1 previous similar message
[ 7669.847629] Lustre: Evicted from MGS (at 192.168.202.140@tcp) after server handle changed from 0x676b594aef763b77 to 0x676b594aef85ddfd
[ 7669.859113] Lustre: Skipped 1 previous similar message
[ 7669.864626] Lustre: MGC192.168.202.140@tcp: Connection restored to 192.168.202.140@tcp (at 192.168.202.140@tcp)
[ 7669.881884] Lustre: Skipped 12 previous similar messages
[ 7707.967132] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 10:01:10 (1788530470)
[ 7709.662725] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7712.158664] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 10:01:14 (1788530474)
[ 7715.112620] LustreError: lustre-MDT0000-mdc-ffff923f83c16800: operation ldlm_enqueue to node 192.168.202.140@tcp failed: rc = -107
[ 7715.132484] LustreError: Skipped 25 previous similar messages
[ 7715.145208] Lustre: lustre-MDT0000-mdc-ffff923f83c16800: Connection to lustre-MDT0000 (at 192.168.202.140@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7715.170575] Lustre: Skipped 1 previous similar message
[ 7715.198621] LustreError: lustre-MDT0000-mdc-ffff923f83c16800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7724.951861] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 10:01:27 (1788530487)
[ 7892.774973] Lustre: Evicted from lustre-OST0000_UUID (at 192.168.202.140@tcp) after server handle changed from 0x676b594aef85e202 to 0x676b594aef85e30c
[ 7892.786494] LustreError: lustre-OST0000-osc-ffff923f83c16800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7895.231129] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 10:04:17 (1788530657)
[ 7922.496591] Lustre: lustre-OST0000-osc-ffff923f83c16800: Connection restored to 192.168.202.140@tcp (at 192.168.202.140@tcp)
[ 7922.508946] Lustre: Skipped 2 previous similar messages
[ 7940.324838] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 10:05:03 (1788530703)
[ 7942.058499] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7944.475791] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 10:05:07 (1788530707)
[ 7946.194300] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7948.170658] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 10:05:10 (1788530710)
[ 7985.694881] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 10:05:48 (1788530748)
[ 8026.090758] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 192.168.202.140@tcp) was lost; in progress operations using this service will fail
[ 8026.134924] Lustre: Evicted from MGS (at 192.168.202.140@tcp) after server handle changed from 0x676b594aef85ddfd to 0x676b594aef860659
[ 8043.435163] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 10:06:46 (1788530806)
[ 8045.533454] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 8047.258529] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 10:06:50 (1788530810)
[ 8049.117735] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 8051.135363] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 10:06:53 (1788530813)
[ 8062.585613] Lustre: Unmounted lustre-client
[ 8065.459522] Lustre: Mounted lustre-client - version 2.17.58_2_gdacfdd2
[ 8069.653544] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 8092.127983] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 192.168.202.140@tcp) was lost; in progress operations using this service will fail
[ 8102.389214] Lustre: Evicted from MGS (at 192.168.202.140@tcp) after server handle changed from 0x676b594aef861031 to 0x676b594aef861085
[ 8102.444353] Lustre: MGC192.168.202.140@tcp: Connection restored to 192.168.202.140@tcp (at 192.168.202.140@tcp)
[ 8102.477569] Lustre: Skipped 2 previous similar messages
[ 8115.221646] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2