[ 1861.793576] LustreError: lustre-OST0000-osc-ffff8b290a1e7800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1861.820864] Lustre: 2354:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.158@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1869.660734] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 09:20:20 (1788614420)
[ 1871.767330] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1873.587828] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 09:20:24 (1788614424)
[ 1875.187035] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1876.819698] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 09:20:27 (1788614427)
[ 1880.087126] LustreError: lustre-MDT0000-mdc-ffff8b290a1e7800: operation mds_reint to node 192.168.203.158@tcp failed: rc = -19
[ 1880.087126] LustreError: lustre-MDT0000-mdc-ffff8b290a1e7800: operation mds_close to node 192.168.203.158@tcp failed: rc = -19
[ 1880.087137] LustreError: Skipped 1 previous similar message
[ 1897.953645] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 192.168.203.158@tcp) was lost; in progress operations using this service will fail
[ 1908.144863] Lustre: Evicted from MGS (at 192.168.203.158@tcp) after server handle changed from 0x2904ef2fcf597863 to 0x2904ef2fcf599367
[ 2008.974450] LustreError: lustre-MDT0000-mdc-ffff8b290a1e7800: operation mds_close to node 192.168.203.158@tcp failed: rc = -19
[ 2008.997896] LustreError: Skipped 2 previous similar messages
[ 2024.927377] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 192.168.203.158@tcp) was lost; in progress operations using this service will fail
[ 2035.115977] Lustre: Evicted from MGS (at 192.168.203.158@tcp) after server handle changed from 0x2904ef2fcf599367 to 0x2904ef2fcf5b9bd6
[ 2050.665862] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 09:23:21 (1788614601)
[ 2051.027663] Lustre: *** cfs_fail_loc=305, val=0***
[ 2051.034877] Lustre: Skipped 4 previous similar messages
[ 2091.495346] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 192.168.203.158@tcp) was lost; in progress operations using this service will fail
[ 2100.820836] Lustre: Evicted from MGS (at 192.168.203.158@tcp) after server handle changed from 0x2904ef2fcf5b9bd6 to 0x2904ef2fcf5ba26d
[ 2107.165382] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2109.030722] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2118.758942] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 09:24:29 (1788614669)
[ 2131.430102] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 192.168.203.158@tcp) was lost; in progress operations using this service will fail
[ 2131.451261] Lustre: Evicted from MGS (at 192.168.203.158@tcp) after server handle changed from 0x2904ef2fcf5ba26d to 0x2904ef2fcf5ba323
[ 2133.528394] LustreError: lustre-MDT0000-mdc-ffff8b290a1e7800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2156.581862] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 09:25:07 (1788614707)
[ 2172.402329] LustreError: lustre-OST0000-osc-ffff8b290a1e7800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2191.494878] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 09:25:42 (1788614742)
[ 2204.626695] LustreError: lustre-MDT0000-mdc-ffff8b290a1e7800: operation mds_reint to node 192.168.203.158@tcp failed: rc = -19
[ 2224.607414] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 192.168.203.158@tcp) was lost; in progress operations using this service will fail
[ 2224.650528] Lustre: Evicted from MGS (at 192.168.203.158@tcp) after server handle changed from 0x2904ef2fcf5ba323 to 0x2904ef2fcf5bec56
[ 2226.036699] Lustre: 15934:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.158@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2253.205794] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2256.001418] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2341.347620] Lustre: 2353:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788614877/real 1788614877]  req@ffff8b2905e61880 x1875496243067648/t0(0) o400->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 224/224 e 0 to 1 dl 1788614893 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2341.401964] Lustre: 2353:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 13 previous similar messages
[ 2341.428957] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 192.168.203.158@tcp) was lost; in progress operations using this service will fail
[ 2351.656737] Lustre: Evicted from MGS (at 192.168.203.158@tcp) after server handle changed from 0x2904ef2fcf5bec56 to 0x2904ef2fcf5d6a0e
[ 2368.058809] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2370.824873] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2437.044633] LustreError: lustre-MDT0000-mdc-ffff8b290a1e7800: operation ldlm_enqueue to node 192.168.203.158@tcp failed: rc = -19
[ 2437.071787] LustreError: Skipped 1 previous similar message
[ 2456.991988] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 192.168.203.158@tcp) was lost; in progress operations using this service will fail
[ 2467.189853] Lustre: Evicted from MGS (at 192.168.203.158@tcp) after server handle changed from 0x2904ef2fcf5d6a0e to 0x2904ef2fcf5f0111
[ 2467.208858] Lustre: MGC192.168.203.158@tcp: Connection restored to 192.168.203.158@tcp (at 192.168.203.158@tcp)
[ 2467.220179] Lustre: Skipped 15 previous similar messages
[ 2485.878562] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2488.581924] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2518.825415] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 09:31:09 (1788615069)
[ 2524.272024] Lustre: lustre-MDT0000-mdc-ffff8b290a1e7800: Connection to lustre-MDT0000 (at 192.168.203.158@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2524.303937] Lustre: Skipped 9 previous similar messages
[ 2543.007691] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 192.168.203.158@tcp) was lost; in progress operations using this service will fail
[ 2553.190676] Lustre: Evicted from MGS (at 192.168.203.158@tcp) after server handle changed from 0x2904ef2fcf5f0111 to 0x2904ef2fcf5fd882
[ 2560.159419] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2591.863049] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2638.006556] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2680.120693] Lustre: 15934:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.158@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2694.589767] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2718.275734] LustreError: lustre-MDT0000-mdc-ffff8b290a1e7800: operation mds_reint to node 192.168.203.158@tcp failed: rc = -19
[ 2718.282484] LustreError: Skipped 5 previous similar messages
[ 2739.167343] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 192.168.203.158@tcp) was lost; in progress operations using this service will fail
[ 2739.184389] LustreError: Skipped 3 previous similar messages
[ 2739.210545] Lustre: Evicted from MGS (at 192.168.203.158@tcp) after server handle changed from 0x2904ef2fcf5ff681 to 0x2904ef2fcf60a0fe
[ 2739.219159] Lustre: Skipped 3 previous similar messages
[ 2740.974561] Lustre: 15934:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.158@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2761.306313] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2821.232508] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2877.697047] Lustre: 15934:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.158@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2918.925740] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 09:37:49 (1788615469)
[ 2972.082451] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 2974.180765] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3264.250706] LustreError: lustre-OST0000-osc-ffff8b290a1e7800: operation ldlm_enqueue to node 192.168.203.158@tcp failed: rc = -107
[ 3264.263662] LustreError: Skipped 5 previous similar messages
[ 3264.270735] Lustre: lustre-OST0000-osc-ffff8b290a1e7800: Connection to lustre-OST0000 (at 192.168.203.158@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3264.283653] Lustre: Skipped 6 previous similar messages
[ 3289.911827] Lustre: lustre-OST0000-osc-ffff8b290a1e7800: Connection restored to 192.168.203.158@tcp (at 192.168.203.158@tcp)
[ 3289.935615] Lustre: Skipped 15 previous similar messages
[ 3314.252494] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3317.154195] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3640.066794] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3643.170177] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3891.100276] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 09:54:02 (1788616442)
[ 3909.087231] Lustre: 50590:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788616445/real 1788616445]  req@ffff8b2906675c00 x1875496264037888/t0(0) o101->lustre-MDT0000-mdc-ffff8b290a1e7800@192.168.203.158@tcp:12/10 lens 576/1152 e 0 to 1 dl 1788616461 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3909.126090] Lustre: 50590:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
[ 3909.135085] Lustre: lustre-MDT0000-mdc-ffff8b290a1e7800: Connection to lustre-MDT0000 (at 192.168.203.158@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3909.161365] Lustre: Skipped 1 previous similar message
[ 3909.193734] Lustre: lustre-MDT0000-mdc-ffff8b290a1e7800: Connection restored to 192.168.203.158@tcp (at 192.168.203.158@tcp)
[ 3909.201239] Lustre: Skipped 1 previous similar message
[ 3916.602576] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 09:54:27 (1788616467)
[ 3941.454391] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 09:54:52 (1788616492)
[ 3965.241285] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 09:55:16 (1788616516)
[ 3965.637818] Lustre: Mounted lustre-client - version 2.17.58_2_ge9089f3
[ 3996.660261] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 192.168.203.158@tcp) was lost; in progress operations using this service will fail
[ 3996.670669] LustreError: Skipped 2 previous similar messages
[ 3996.688526] Lustre: Evicted from MGS (at 192.168.203.158@tcp) after server handle changed from 0x2904ef2fcf61d7e4 to 0x2904ef2fcf760956
[ 3996.702346] Lustre: Skipped 2 previous similar messages
[ 3997.663319] Lustre: 2354:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788616533/real 1788616533]  req@ffff8b2906cf4700 x1875496264057088/t0(0) o400->lustre-MDT0000-mdc-ffff8b290a1e7800@192.168.203.158@tcp:12/10 lens 224/224 e 0 to 1 dl 1788616549 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3997.702363] Lustre: 2354:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 4008.458784] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4010.037623] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4012.284430] Lustre: Unmounted lustre-client
[ 4018.620366] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 09:56:09 (1788616569)
[ 4156.383355] Lustre: 2354:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788616692/real 1788616692]  req@ffff8b2906674a80 x1875496264090368/t0(0) o4->lustre-OST0000-osc-ffff8b290a1e7800@192.168.203.158@tcp:6/4 lens 488/448 e 0 to 1 dl 1788616708 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4156.437537] Lustre: 2354:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 68 previous similar messages
[ 4314.897556] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 10:01:05 (1788616865)
[ 4364.470522] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 10:01:55 (1788616915)
[ 4368.167198] Lustre: Unmounted lustre-client
[ 4389.880865] Lustre: Mounted lustre-client - version 2.17.58_2_ge9089f3
[ 4396.553484] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 10:02:27 (1788616947)
[ 4397.174444] LustreError: 56522:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4398.225049] LustreError: 56522:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4398.454486] Lustre: *** cfs_fail_loc=305, val=0***
[ 4423.168467] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 10:02:54 (1788616974)
[ 4423.631854] Lustre: Mounted lustre-client - version 2.17.58_2_ge9089f3
[ 4426.119188] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4436.429157] Lustre: Unmounted lustre-client
[ 4444.680668] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 10:03:15 (1788616995)
[ 4540.358120] LustreError: lustre-MDT0000-mdc-ffff8b2903085000: operation mds_reint to node 192.168.203.158@tcp failed: rc = -19
[ 4540.372940] LustreError: Skipped 1 previous similar message
[ 4540.375617] Lustre: lustre-MDT0000-mdc-ffff8b2903085000: Connection to lustre-MDT0000 (at 192.168.203.158@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4540.385354] Lustre: Skipped 23 previous similar messages
[ 4561.887569] Lustre: 2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788617098/real 1788617098]  req@ffff8b2906674380 x1875496266769920/t0(0) o400->MGC192.168.203.158@tcp@192.168.203.158@tcp:26/25 lens 224/224 e 0 to 1 dl 1788617114 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4561.898657] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 192.168.203.158@tcp) was lost; in progress operations using this service will fail
[ 4561.909556] Lustre: 2352:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 87 previous similar messages
[ 4561.928118] Lustre: Evicted from MGS (at 192.168.203.158@tcp) after server handle changed from 0x2904ef2fcf76208d to 0x2904ef2fcf78d460
[ 4561.939958] Lustre: MGC192.168.203.158@tcp: Connection restored to 192.168.203.158@tcp (at 192.168.203.158@tcp)
[ 4561.947253] Lustre: Skipped 24 previous similar messages
[ 4563.753381] Lustre: 55920:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.158@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 4756.919557] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 10:08:27 (1788617307)
[ 4762.298957] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4775.921138] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 192.168.203.158@tcp) was lost; in progress operations using this service will fail
[ 4775.936163] Lustre: Evicted from MGS (at 192.168.203.158@tcp) after server handle changed from 0x2904ef2fcf78d460 to 0x2904ef2fcf7d9d3d
[ 4786.236637] LustreError: lustre-MDT0000-mdc-ffff8b2903085000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4794.986865] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 10:09:05 (1788617345)
[ 4795.508944] Lustre: Mounted lustre-client - version 2.17.58_2_ge9089f3
[ 4803.149948] LustreError: lustre-OST0000-osc-ffff8b2903085000: operation ldlm_enqueue to node 192.168.203.158@tcp failed: rc = -107
[ 4803.192649] LustreError: lustre-OST0000-osc-ffff8b2903085000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4816.939912] Lustre: Unmounted lustre-client
[ 4827.221697] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 10:09:37 (1788617377)
[ 4834.691731] LustreError: lustre-MDT0000-mdc-ffff8b2903085000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4834.708523] LustreError: 60921:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4842.016975] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 10:09:53 (1788617393)
[ 4842.304595] LustreError: 61626:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4846.517698] LustreError: lustre-MDT0000-mdc-ffff8b2903085000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4846.544325] LustreError: 61643:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4846.557063] LustreError: 61626:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=751
[ 4846.571620] LustreError: 61626:0:(import.c:761:ptlrpc_connect_import_locked()) already connecting
[ 4846.625641] LustreError: 61648:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4847.767919] LustreError: 61654:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4847.781916] LustreError: 61654:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4850.182476] LustreError: 61676:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4850.199311] LustreError: 61676:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4854.944842] LustreError: 61721:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4854.963090] LustreError: 61721:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4865.014169] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 10:10:15 (1788617415)
[ 4904.321308] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4906.134522] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4917.384651] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 10:11:08 (1788617468)
[ 4955.127833] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4957.057747] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4968.469513] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 10:11:59 (1788617519)
[ 5032.426689] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5034.323924] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5043.639243] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 10:13:14 (1788617594)
[ 5080.901379] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5082.659504] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5088.946725] Lustre: Unmounted lustre-client
[ 5144.902934] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5148.035574] Lustre: Mounted lustre-client - version 2.17.58_2_ge9089f3
[ 5155.574488] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 10:15:06 (1788617706)
[ 5158.248757] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5160.335732] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 10:15:10 (1788617710)
[ 5193.991547] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 10:15:44 (1788617744)
[ 5195.718906] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5197.997956] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 10:15:48 (1788617748)
[ 5199.611290] Lustre: *** cfs_fail_loc=805, val=0***
[ 5199.646539] Lustre: Mounted lustre-client - version 2.17.58_2_ge9089f3
[ 5204.813702] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5225.449467] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 192.168.203.158@tcp) was lost; in progress operations using this service will fail
[ 5225.495759] Lustre: Evicted from MGS (at 192.168.203.158@tcp) after server handle changed from 0x2904ef2fcf7da96f to 0x2904ef2fcf7dab8a
[ 5225.519915] Lustre: MGC192.168.203.158@tcp: Connection restored to 192.168.203.158@tcp (at 192.168.203.158@tcp)
[ 5225.534116] Lustre: Skipped 12 previous similar messages
[ 5226.463312] Lustre: 2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788617762/real 1788617762]  req@ffff8b2906d9b480 x1875496268824320/t0(0) o400->lustre-MDT0000-mdc-ffff8b290a1e2000@192.168.203.158@tcp:12/10 lens 224/224 e 0 to 1 dl 1788617778 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5226.510301] Lustre: 2352:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5226.522650] Lustre: lustre-MDT0000-mdc-ffff8b290a1e2000: Connection to lustre-MDT0000 (at 192.168.203.158@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5226.537460] Lustre: Skipped 12 previous similar messages
[ 5227.313180] Lustre: 68287:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.158@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 5243.352095] Lustre: Unmounted lustre-client
[ 5251.789887] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 10:16:42 (1788617802)
[ 5279.590218] LustreError: 2350:0:(client.c:3449:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff8b2906d9bb80 x1875496268836608/t90194313221(90194313221) o101->lustre-MDT0000-mdc-ffff8b2907243000@192.168.203.158@tcp:12/10 lens 648/608 e 0 to 0 dl 1788617847 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5289.706762] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5291.355706] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5300.081789] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 10:17:30 (1788617850)
[ 5301.802277] LustreError: lustre-OST0000-osc-ffff8b2907243000: operation ost_write to node 192.168.203.158@tcp failed: rc = -107
[ 5301.814199] LustreError: Skipped 2 previous similar messages
[ 5301.828391] LustreError: lustre-OST0000-osc-ffff8b2907243000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5301.840553] Lustre: 2353:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.158@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5301.859815] LustreError: 72827:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff8b2907243000: namespace resource [0x240000400:0x3c02:0x0].0x0 (ffff8b29108b5000) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5310.339720] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 10:17:41 (1788617861)
[ 5311.983994] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5314.249217] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 10:17:44 (1788617864)
[ 5316.036200] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5317.610118] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 10:17:48 (1788617868)
[ 5319.385726] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5321.670065] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 10:17:52 (1788617872)
[ 5323.433534] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5325.266814] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 10:17:56 (1788617876)
[ 5326.902824] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5328.975185] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 10:17:59 (1788617879)
[ 5330.651284] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5332.117908] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 10:18:03 (1788617883)
[ 5333.765969] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5335.465984] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 10:18:06 (1788617886)
[ 5337.036951] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5338.803669] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 10:18:09 (1788617889)
[ 5340.196444] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5342.043047] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 10:18:12 (1788617892)
[ 5343.338911] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5344.906526] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 10:18:16 (1788617896)
[ 5346.286755] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5348.123846] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 10:18:19 (1788617899)
[ 5350.842659] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5352.590946] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 10:18:23 (1788617903)
[ 5374.951907] Lustre: 68287:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.158@tcp: IR log lustre-cliir failed, not fatal: rc = -5
[ 5384.117959] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 10:18:54 (1788617934)
[ 5414.532537] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 10:19:25 (1788617965)
[ 5415.105769] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5425.632198] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 10:19:36 (1788617976)
[ 5426.252907] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5437.031335] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 10:19:47 (1788617987)
[ 5437.577414] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5445.964903] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 10:19:56 (1788617996)
[ 5446.325230] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5454.973505] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 10:20:05 (1788618005)
[ 5455.389518] Lustre: *** cfs_fail_loc=510, val=3***
[ 5464.046933] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 10:20:15 (1788618015)
[ 5464.460491] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5475.897632] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 10:20:26 (1788618026)
[ 5476.280571] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5542.880495] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 10:21:33 (1788618093)
[ 5543.447835] LustreError: 83035:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5546.527443] LustreError: lustre-MDT0000-mdc-ffff8b2907243000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5546.547369] LustreError: 83049:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8b2907243000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8b2908943d00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5547.543161] LustreError: 83035:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5549.655418] LustreError: 83065:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5552.584774] LustreError: lustre-MDT0000-mdc-ffff8b2907243000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5552.614696] LustreError: 83079:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8b2907243000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8b29108b5600) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5552.658903] LustreError: 83079:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5553.751121] LustreError: 83065:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5560.781308] LustreError: lustre-MDT0000-mdc-ffff8b2907243000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5561.304323] LustreError: 83121:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5564.546091] LustreError: lustre-MDT0000-mdc-ffff8b2907243000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5564.611474] LustreError: 83141:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5564.619591] LustreError: 83141:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5565.383439] LustreError: 83121:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5565.388340] LustreError: 83121:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8b2907243000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5565.674079] LustreError: 83147:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5565.690727] LustreError: 83147:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5568.085989] LustreError: 83170:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5568.100748] LustreError: 83170:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5572.086248] LustreError: 83196:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5576.087309] LustreError: 83196:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5576.323945] LustreError: lustre-MDT0000-mdc-ffff8b2907243000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5583.521434] LustreError: lustre-MDT0000-mdc-ffff8b2907243000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5584.146469] LustreError: 83253:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5587.659162] LustreError: lustre-MDT0000-mdc-ffff8b2907243000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5587.676660] LustreError: 83267:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8b2907243000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8b2908943f00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5587.713578] LustreError: 83267:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5588.231386] LustreError: 83253:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5591.376112] LustreError: lustre-MDT0000-mdc-ffff8b2907243000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5591.506162] LustreError: 83300:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5591.523021] LustreError: 83300:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5592.480796] LustreError: 83280:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8b2907243000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5600.611073] LustreError: 83352:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5600.623833] LustreError: 83352:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5603.974532] LustreError: lustre-MDT0000-mdc-ffff8b2907243000: operation ldlm_enqueue to node 192.168.203.158@tcp failed: rc = -107
[ 5603.987878] LustreError: Skipped 8 previous similar messages
[ 5604.005343] LustreError: lustre-MDT0000-mdc-ffff8b2907243000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5604.098758] LustreError: 83375:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5604.107349] LustreError: 83375:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5604.703173] LustreError: 83352:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5604.709425] LustreError: 83352:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5604.718641] LustreError: 83352:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8b2907243000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5616.089120] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 10:22:46 (1788618166)
[ 5634.842867] LustreError: 79786:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5634.851253] LustreError: 79786:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5638.927116] LustreError: 79786:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5638.939326] LustreError: 79786:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5646.242760] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 10:23:17 (1788618197)
[ 5695.941323] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 10:24:06 (1788618246)
[ 5760.043808] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 10:25:10 (1788618310)
[ 5794.595212] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 10:25:45 (1788618345)
[ 5795.159699] Lustre: Mounted lustre-client - version 2.17.58_2_ge9089f3
[ 5918.590231] Lustre: Unmounted lustre-client
[ 5925.436668] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 10:27:56 (1788618476)
[ 5925.945866] LustreError: 87684:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5928.438653] Lustre: lustre-OST0000-osc-ffff8b2907243000: Connection to lustre-OST0000 (at 192.168.203.158@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5928.482103] Lustre: Skipped 17 previous similar messages
[ 5928.525332] LustreError: lustre-OST0000-osc-ffff8b2907243000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5928.553604] LustreError: 87771:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff8b2907243000: namespace resource [0x240000400:0x3c2d:0x0].0x0 (ffff8b29108b5700) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5928.582154] LustreError: 87771:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5928.831166] LustreError: 87684:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5928.834778] Lustre: 2351:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.158@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5928.846409] Lustre: lustre-OST0000-osc-ffff8b2907243000: Connection restored to 192.168.203.158@tcp (at 192.168.203.158@tcp)
[ 5928.853370] Lustre: Skipped 19 previous similar messages
[ 5936.677279] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 10:28:07 (1788618487)
[ 5995.490825] Lustre: 88352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788618492/real 1788618492]  req@ffff8b2905f86300 x1875496269005824/t0(0) o101->lustre-MDT0000-mdc-ffff8b2907243000@192.168.203.158@tcp:12/10 lens 328/344 e 0 to 1 dl 1788618547 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 5995.532917] Lustre: 88352:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 18 previous similar messages
[ 6003.854482] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 10:29:14 (1788618554)
[ 6005.584330] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 6007.523991] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 10:29:18 (1788618558)
[ 6071.478498] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6073.478908] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 10:30:24 (1788618624)
[ 6132.216200] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 10:31:23 (1788618683)
[ 6134.899402] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6136.756372] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 10:31:27 (1788618687)
[ 6138.250717] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6140.379267] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 10:31:31 (1788618691)
[ 6174.208313] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 10:32:05 (1788618725)
[ 6186.168926] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6213.343295] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 192.168.203.158@tcp) was lost; in progress operations using this service will fail
[ 6213.355785] LustreError: Skipped 3 previous similar messages
[ 6222.891886] Lustre: Evicted from MGS (at 192.168.203.158@tcp) after server handle changed from 0x2904ef2fcf7db7d1 to 0x2904ef2fcf7dd273
[ 6222.903675] Lustre: Skipped 3 previous similar messages
[ 6229.623079] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6231.762695] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6243.479351] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 10:33:14 (1788618794)
[ 6246.795303] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6248.464485] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 10:33:19 (1788618799)
[ 6263.783764] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 192.168.203.158@tcp) was lost; in progress operations using this service will fail
[ 6263.828364] Lustre: Evicted from MGS (at 192.168.203.158@tcp) after server handle changed from 0x2904ef2fcf7dd273 to 0x2904ef2fcf7dd48e
[ 6279.501327] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 10:33:49 (1788618829)
[ 6311.925470] Lustre: 68287:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.158@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6331.552969] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 10:34:42 (1788618882)
[ 6338.584174] LustreError: lustre-OST0000-osc-ffff8b2907243000: operation ost_setattr to node 192.168.203.158@tcp failed: rc = -19
[ 6338.593928] LustreError: Skipped 10 previous similar messages
[ 6404.663862] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6409.254043] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6495.199424] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 192.168.203.158@tcp) was lost; in progress operations using this service will fail
[ 6495.213743] LustreError: Skipped 1 previous similar message
[ 6505.447786] Lustre: Evicted from MGS (at 192.168.203.158@tcp) after server handle changed from 0x2904ef2fcf7dd69b to 0x2904ef2fcf7de4f6
[ 6505.463382] Lustre: Skipped 1 previous similar message
[ 6521.075676] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6523.303753] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6547.438388] Lustre: lustre-MDT0000-mdc-ffff8b2907243000: Connection to lustre-MDT0000 (at 192.168.203.158@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6547.445480] Lustre: MGC192.168.203.158@tcp: Connection restored to 192.168.203.158@tcp (at 192.168.203.158@tcp)
[ 6547.463573] Lustre: Skipped 8 previous similar messages
[ 6547.489278] Lustre: Skipped 11 previous similar messages
[ 6563.159617] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6564.837723] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6596.575254] Lustre: 2353:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788619088/real 1788619088]  req@ffff8b2938843800 x1875496275544832/t0(0) o400->lustre-MDT0000-mdc-ffff8b2907243000@192.168.203.158@tcp:12/10 lens 224/224 e 0 to 1 dl 1788619148 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6596.616634] Lustre: 2353:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 26 previous similar messages
[ 6730.609696] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 10:41:21 (1788619281)
[ 6802.120202] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6806.426347] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 7082.127933] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 10:47:12 (1788619632)
[ 7252.966849] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 192.168.203.158@tcp) was lost; in progress operations using this service will fail
[ 7252.984967] LustreError: Skipped 1 previous similar message
[ 7252.989767] Lustre: lustre-MDT0000-mdc-ffff8b2907243000: Connection to lustre-MDT0000 (at 192.168.203.158@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7253.001755] Lustre: Evicted from MGS (at 192.168.203.158@tcp) after server handle changed from 0x2904ef2fcf7de837 to 0x2904ef2fcf8d8b9d
[ 7253.008087] Lustre: Skipped 1 previous similar message
[ 7253.025205] Lustre: Skipped 1 previous similar message
[ 7253.034959] Lustre: MGC192.168.203.158@tcp: Connection restored to 192.168.203.158@tcp (at 192.168.203.158@tcp)
[ 7253.045963] Lustre: Skipped 1 previous similar message
[ 7280.097653] Lustre: 2353:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788619795/real 1788619795]  req@ffff8b29105c6680 x1875496286703744/t0(0) o400->lustre-MDT0000-mdc-ffff8b2907243000@192.168.203.158@tcp:12/10 lens 224/224 e 0 to 1 dl 1788619832 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7280.119845] Lustre: 2353:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 7286.918496] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 10:50:37 (1788619837)
[ 7289.140036] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7290.880757] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 10:50:41 (1788619841)
[ 7292.947355] LustreError: lustre-MDT0000-mdc-ffff8b2907243000: operation ldlm_enqueue to node 192.168.203.158@tcp failed: rc = -107
[ 7292.960837] LustreError: Skipped 100 previous similar messages
[ 7292.983938] LustreError: lustre-MDT0000-mdc-ffff8b2907243000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7300.989271] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 10:50:51 (1788619851)
[ 7468.715931] Lustre: Evicted from lustre-OST0000_UUID (at 192.168.203.158@tcp) after server handle changed from 0x2904ef2fcf8d90f9 to 0x2904ef2fcf8d9203
[ 7468.727220] LustreError: lustre-OST0000-osc-ffff8b2907243000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7470.539794] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 10:53:41 (1788620021)
[ 7513.187917] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 10:54:23 (1788620063)
[ 7514.678322] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7516.658322] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 10:54:27 (1788620067)
[ 7518.226911] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7519.690156] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 10:54:30 (1788620070)
[ 7555.302746] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 10:55:05 (1788620105)
[ 7597.040702] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 192.168.203.158@tcp) was lost; in progress operations using this service will fail
[ 7597.079400] Lustre: Evicted from MGS (at 192.168.203.158@tcp) after server handle changed from 0x2904ef2fcf8d8b9d to 0x2904ef2fcf8db550
[ 7614.680503] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 10:56:05 (1788620165)
[ 7616.486901] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7618.250779] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 10:56:09 (1788620169)
[ 7620.025731] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7621.885173] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 10:56:12 (1788620172)
[ 7633.343688] Lustre: Unmounted lustre-client
[ 7635.230143] Lustre: Mounted lustre-client - version 2.17.58_2_ge9089f3
[ 7638.329768] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7656.929826] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 192.168.203.158@tcp) was lost; in progress operations using this service will fail
[ 7667.125712] Lustre: Evicted from MGS (at 192.168.203.158@tcp) after server handle changed from 0x2904ef2fcf8dbe48 to 0x2904ef2fcf8dc1e4
[ 7672.260613] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2