[ 1843.000762] LustreError: lustre-OST0000-osc-ffff93fa42feb000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1843.019365] Lustre: 2354:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.206.114@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1848.987476] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 10:56:13 (1787583373)
[ 1851.201974] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1852.679237] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 10:56:16 (1787583376)
[ 1854.324538] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1856.158334] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 10:56:20 (1787583380)
[ 1859.516100] LustreError: lustre-MDT0000-mdc-ffff93fa42feb000: operation ldlm_enqueue to node 192.168.206.114@tcp failed: rc = -19
[ 1859.533046] LustreError: Skipped 1 previous similar message
[ 1877.471362] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 192.168.206.114@tcp) was lost; in progress operations using this service will fail
[ 1886.636170] Lustre: Evicted from MGS (at 192.168.206.114@tcp) after server handle changed from 0x37be1cabc7732308 to 0x37be1cabc7733fc5
[ 1980.249438] LustreError: lustre-MDT0000-mdc-ffff93fa42feb000: operation mds_close to node 192.168.206.114@tcp failed: rc = -19
[ 1980.254883] LustreError: Skipped 1 previous similar message
[ 1999.135458] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 192.168.206.114@tcp) was lost; in progress operations using this service will fail
[ 1999.170353] Lustre: Evicted from MGS (at 192.168.206.114@tcp) after server handle changed from 0x37be1cabc7733fc5 to 0x37be1cabc7751de0
[ 2000.881045] Lustre: 15914:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.114@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2022.799108] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 10:59:07 (1787583547)
[ 2023.020407] Lustre: *** cfs_fail_loc=305, val=0***
[ 2023.022909] Lustre: Skipped 2 previous similar messages
[ 2063.839208] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 192.168.206.114@tcp) was lost; in progress operations using this service will fail
[ 2063.868071] Lustre: Evicted from MGS (at 192.168.206.114@tcp) after server handle changed from 0x37be1cabc7751de0 to 0x37be1cabc77523d6
[ 2065.905157] Lustre: 15914:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.114@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2087.298279] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2088.881304] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2097.373955] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 11:00:21 (1787583621)
[ 2109.928083] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 192.168.206.114@tcp) was lost; in progress operations using this service will fail
[ 2109.940299] Lustre: lustre-MDT0000-mdc-ffff93fa42feb000: Connection to lustre-MDT0000 (at 192.168.206.114@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2109.957210] Lustre: Skipped 12 previous similar messages
[ 2109.968692] Lustre: Evicted from MGS (at 192.168.206.114@tcp) after server handle changed from 0x37be1cabc77523d6 to 0x37be1cabc7752955
[ 2110.063478] LustreError: lustre-MDT0000-mdc-ffff93fa42feb000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2132.061181] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 11:00:55 (1787583655)
[ 2150.383681] LustreError: lustre-OST0000-osc-ffff93fa42feb000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2150.396059] Lustre: lustre-OST0000-osc-ffff93fa42feb000: Connection restored to 192.168.206.114@tcp (at 192.168.206.114@tcp)
[ 2150.401704] Lustre: Skipped 16 previous similar messages
[ 2163.375765] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 11:01:27 (1787583687)
[ 2176.614712] LustreError: lustre-MDT0000-mdc-ffff93fa42feb000: operation mds_reint to node 192.168.206.114@tcp failed: rc = -19
[ 2196.506843] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 192.168.206.114@tcp) was lost; in progress operations using this service will fail
[ 2196.528532] Lustre: Evicted from MGS (at 192.168.206.114@tcp) after server handle changed from 0x37be1cabc7752955 to 0x37be1cabc7757ac3
[ 2215.297186] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2217.339482] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2298.847159] Lustre: 2352:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787583808/real 1787583808]  req@ffff93fa48501880 x1874415155195392/t0(0) o400->MGC192.168.206.114@tcp@192.168.206.114@tcp:26/25 lens 224/224 e 0 to 1 dl 1787583824 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2298.879264] Lustre: 2352:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 10 previous similar messages
[ 2298.888724] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 192.168.206.114@tcp) was lost; in progress operations using this service will fail
[ 2309.106919] Lustre: Evicted from MGS (at 192.168.206.114@tcp) after server handle changed from 0x37be1cabc7757ac3 to 0x37be1cabc77764a7
[ 2327.205497] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2329.462314] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2394.554799] LustreError: lustre-MDT0000-mdc-ffff93fa42feb000: operation mds_close to node 192.168.206.114@tcp failed: rc = -19
[ 2394.563501] LustreError: Skipped 3 previous similar messages
[ 2411.487303] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 192.168.206.114@tcp) was lost; in progress operations using this service will fail
[ 2420.652442] Lustre: Evicted from MGS (at 192.168.206.114@tcp) after server handle changed from 0x37be1cabc77764a7 to 0x37be1cabc7792732
[ 2436.239776] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2438.061184] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2466.393942] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 11:06:30 (1787583990)
[ 2487.263635] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 192.168.206.114@tcp) was lost; in progress operations using this service will fail
[ 2497.615388] Lustre: Evicted from MGS (at 192.168.206.114@tcp) after server handle changed from 0x37be1cabc7792732 to 0x37be1cabc77a26ea
[ 2497.975435] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2526.475794] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2557.592343] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2598.859473] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2638.817975] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 192.168.206.114@tcp) was lost; in progress operations using this service will fail
[ 2638.837533] LustreError: Skipped 3 previous similar messages
[ 2649.076413] Lustre: Evicted from MGS (at 192.168.206.114@tcp) after server handle changed from 0x37be1cabc77a7270 to 0x37be1cabc77ac41d
[ 2649.084515] Lustre: Skipped 3 previous similar messages
[ 2655.951687] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2683.766642] LustreError: lustre-MDT0000-mdc-ffff93fa42feb000: operation ldlm_enqueue to node 192.168.206.114@tcp failed: rc = -19
[ 2683.781677] LustreError: Skipped 4 previous similar messages
[ 2712.205352] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2746.478425] Lustre: lustre-MDT0000-mdc-ffff93fa42feb000: Connection to lustre-MDT0000 (at 192.168.206.114@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2746.502120] Lustre: Skipped 9 previous similar messages
[ 2764.793589] Lustre: MGC192.168.206.114@tcp: Connection restored to 192.168.206.114@tcp (at 192.168.206.114@tcp)
[ 2764.797247] Lustre: Skipped 17 previous similar messages
[ 2804.145354] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 11:12:08 (1787584328)
[ 2855.218634] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 2857.332541] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3188.893940] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3190.786313] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3480.495438] LustreError: lustre-OST0000-osc-ffff93fa42feb000: operation ldlm_enqueue to node 192.168.206.114@tcp failed: rc = -107
[ 3480.501287] LustreError: Skipped 3 previous similar messages
[ 3480.506680] Lustre: lustre-OST0000-osc-ffff93fa42feb000: Connection to lustre-OST0000 (at 192.168.206.114@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3480.521714] Lustre: Skipped 2 previous similar messages
[ 3501.316680] Lustre: lustre-OST0000-osc-ffff93fa42feb000: Connection restored to 192.168.206.114@tcp (at 192.168.206.114@tcp)
[ 3501.334088] Lustre: Skipped 3 previous similar messages
[ 3519.751351] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3521.953711] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3775.000472] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 11:28:19 (1787585299)
[ 3791.839142] Lustre: 50584:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787585301/real 1787585301]  req@ffff93fa48896d80 x1874415178794112/t0(0) o101->lustre-MDT0000-mdc-ffff93fa42feb000@192.168.206.114@tcp:12/10 lens 576/1152 e 0 to 1 dl 1787585317 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3791.874080] Lustre: 50584:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
[ 3800.553901] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 11:28:44 (1787585324)
[ 3824.673810] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 11:29:08 (1787585348)
[ 3849.039747] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 11:29:33 (1787585373)
[ 3849.294246] Lustre: Mounted lustre-client - version 2.17.57_80_g3abb7b2
[ 3879.903656] Lustre: 2353:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787585389/real 1787585390]  req@ffff93fa50133100 x1874415178812928/t0(0) o400->MGC192.168.206.114@tcp@192.168.206.114@tcp:26/25 lens 224/224 e 0 to 1 dl 1787585405 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3879.933552] Lustre: 2353:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 3879.942896] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 192.168.206.114@tcp) was lost; in progress operations using this service will fail
[ 3879.959447] LustreError: Skipped 2 previous similar messages
[ 3890.174667] Lustre: Evicted from MGS (at 192.168.206.114@tcp) after server handle changed from 0x37be1cabc77c7c98 to 0x37be1cabc79314dd
[ 3890.182312] Lustre: Skipped 2 previous similar messages
[ 3900.708216] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 3901.934766] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3903.738816] Lustre: Unmounted lustre-client
[ 3908.644244] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 11:30:33 (1787585433)
[ 4044.769335] Lustre: 2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787585554/real 1787585554]  req@ffff93fa4850ea00 x1874415178847232/t0(0) o4->lustre-OST0000-osc-ffff93fa42feb000@192.168.206.114@tcp:6/4 lens 488/448 e 0 to 1 dl 1787585570 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4044.817881] Lustre: 2355:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 70 previous similar messages
[ 4092.895379] Lustre: lustre-OST0000-osc-ffff93fa42feb000: Connection to lustre-OST0000 (at 192.168.206.114@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4092.919690] Lustre: Skipped 15 previous similar messages
[ 4109.118806] Lustre: lustre-OST0000-osc-ffff93fa42feb000: Connection restored to 192.168.206.114@tcp (at 192.168.206.114@tcp)
[ 4109.127459] Lustre: Skipped 17 previous similar messages
[ 4201.516595] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 11:35:25 (1787585725)
[ 4248.658154] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 11:36:12 (1787585772)
[ 4251.583072] Lustre: Unmounted lustre-client
[ 4271.438764] Lustre: Mounted lustre-client - version 2.17.57_80_g3abb7b2
[ 4276.651456] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 11:36:41 (1787585801)
[ 4276.849257] LustreError: 56503:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4277.903084] LustreError: 56503:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4278.029271] Lustre: *** cfs_fail_loc=305, val=0***
[ 4299.726290] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 11:37:04 (1787585824)
[ 4300.128782] Lustre: Mounted lustre-client - version 2.17.57_80_g3abb7b2
[ 4302.182189] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4312.551766] Lustre: Unmounted lustre-client
[ 4319.085551] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 11:37:23 (1787585843)
[ 4416.591539] LustreError: lustre-MDT0000-mdc-ffff93fa47e5a000: operation mds_reint to node 192.168.206.114@tcp failed: rc = -19
[ 4434.403786] Lustre: 2352:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787585943/real 1787585943]  req@ffff93fa466a0700 x1874415181523456/t0(0) o400->MGC192.168.206.114@tcp@192.168.206.114@tcp:26/25 lens 224/224 e 0 to 1 dl 1787585959 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4434.464023] Lustre: 2352:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 79 previous similar messages
[ 4434.498675] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 192.168.206.114@tcp) was lost; in progress operations using this service will fail
[ 4443.693377] Lustre: Evicted from MGS (at 192.168.206.114@tcp) after server handle changed from 0x37be1cabc7932ed7 to 0x37be1cabc795e574
[ 4596.430226] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 11:42:00 (1787586120)
[ 4601.581492] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4617.206306] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 192.168.206.114@tcp) was lost; in progress operations using this service will fail
[ 4617.257058] Lustre: Evicted from MGS (at 192.168.206.114@tcp) after server handle changed from 0x37be1cabc795e574 to 0x37be1cabc79aade1
[ 4617.300642] LustreError: lustre-MDT0000-mdc-ffff93fa47e5a000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4633.033570] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 11:42:37 (1787586157)
[ 4633.473991] Lustre: Mounted lustre-client - version 2.17.57_80_g3abb7b2
[ 4640.643403] LustreError: lustre-OST0000-osc-ffff93fa47e5a000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4652.202877] Lustre: Unmounted lustre-client
[ 4658.959870] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 11:43:03 (1787586183)
[ 4665.504042] LustreError: lustre-MDT0000-mdc-ffff93fa47e5a000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4665.523542] LustreError: 60983:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff93fa47e5a000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff93fa46d17600) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4665.524381] LustreError: 60899:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4672.401026] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 11:43:16 (1787586196)
[ 4672.775948] LustreError: 61603:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4677.546827] LustreError: lustre-MDT0000-mdc-ffff93fa47e5a000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4677.567834] LustreError: 61620:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4677.588324] LustreError: 61603:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=197
[ 4677.608782] LustreError: 61603:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting
[ 4677.673693] LustreError: 61625:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4678.804438] LustreError: 61631:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4678.819197] LustreError: 61631:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4681.177634] LustreError: 61654:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4681.183749] LustreError: 61654:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4685.695623] LustreError: 61699:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4685.712733] LustreError: 61699:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4694.237518] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 11:43:38 (1787586218)
[ 4703.230116] Lustre: lustre-OST0000-osc-ffff93fa47e5a000: Connection to lustre-OST0000 (at 192.168.206.114@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4703.249335] Lustre: Skipped 17 previous similar messages
[ 4731.607399] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4733.105865] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4742.364519] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 11:44:26 (1787586266)
[ 4765.700631] Lustre: lustre-OST0000-osc-ffff93fa47e5a000: Connection restored to 192.168.206.114@tcp (at 192.168.206.114@tcp)
[ 4765.709944] Lustre: Skipped 17 previous similar messages
[ 4778.513593] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4779.985287] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4791.283250] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 11:45:15 (1787586315)
[ 4854.716889] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4856.029135] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4863.207411] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 11:46:27 (1787586387)
[ 4898.714972] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4900.214170] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4906.628593] Lustre: Unmounted lustre-client
[ 4954.486574] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4956.900456] Lustre: Mounted lustre-client - version 2.17.57_80_g3abb7b2
[ 4962.298994] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 11:48:06 (1787586486)
[ 4964.394643] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 4965.697975] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 11:48:10 (1787586490)
[ 4992.516966] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 11:48:36 (1787586516)
[ 4993.709336] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 4995.087752] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 11:48:39 (1787586519)
[ 4996.085948] Lustre: *** cfs_fail_loc=805, val=0***
[ 4996.138765] Lustre: Mounted lustre-client - version 2.17.57_80_g3abb7b2
[ 5000.551462] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5021.663411] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 192.168.206.114@tcp) was lost; in progress operations using this service will fail
[ 5021.691948] Lustre: Evicted from MGS (at 192.168.206.114@tcp) after server handle changed from 0x37be1cabc79ab7f8 to 0x37be1cabc79abb08
[ 5030.413444] Lustre: Unmounted lustre-client
[ 5039.427219] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 11:49:23 (1787586563)
[ 5057.505418] Lustre: 71424:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787586566/real 1787586566]  req@ffff93fa46679180 x1874415183583232/t0(0) o36->lustre-MDT0000-mdc-ffff93fa49dda000@192.168.206.114@tcp:12/10 lens 504/576 e 0 to 1 dl 1787586582 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'mkdir.0' uid:0 gid:0 projid:4294967295
[ 5057.540855] Lustre: 71424:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
[ 5066.771703] LustreError: 2351:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff93fa4667b800 x1874415183582848/t90194313220(90194313220) o101->lustre-MDT0000-mdc-ffff93fa49dda000@192.168.206.114@tcp:12/10 lens 664/608 e 0 to 0 dl 1787586608 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5073.105066] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5074.682962] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5082.400236] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 11:50:06 (1787586606)
[ 5083.570129] LustreError: lustre-OST0000-osc-ffff93fa49dda000: operation ost_write to node 192.168.206.114@tcp failed: rc = -107
[ 5083.577030] LustreError: Skipped 3 previous similar messages
[ 5083.588243] LustreError: lustre-OST0000-osc-ffff93fa49dda000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5083.600637] Lustre: 2355:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.206.114@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5083.623815] LustreError: 72802:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff93fa49dda000: namespace resource [0x240000400:0x4202:0x0].0x0 (ffff93fa46d17d00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5090.469706] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 11:50:14 (1787586614)
[ 5091.743619] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5093.485023] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 11:50:17 (1787586617)
[ 5094.955952] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5096.422770] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 11:50:20 (1787586620)
[ 5097.761943] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5099.174566] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 11:50:23 (1787586623)
[ 5100.497827] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5101.934520] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 11:50:26 (1787586626)
[ 5103.228592] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5104.730527] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 11:50:28 (1787586628)
[ 5105.952124] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5107.431361] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 11:50:31 (1787586631)
[ 5108.652331] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5110.288202] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 11:50:34 (1787586634)
[ 5111.514312] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5113.080539] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 11:50:37 (1787586637)
[ 5114.137825] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5115.515335] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 11:50:39 (1787586639)
[ 5116.747994] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5118.004784] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 11:50:42 (1787586642)
[ 5119.225866] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5120.387293] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 11:50:44 (1787586644)
[ 5122.339349] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5123.658000] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 11:50:48 (1787586648)
[ 5153.300678] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 11:51:17 (1787586677)
[ 5181.303594] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 11:51:45 (1787586705)
[ 5181.631381] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5191.158215] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 11:51:55 (1787586715)
[ 5191.598673] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5203.086333] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 11:52:06 (1787586726)
[ 5203.481899] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5213.124921] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 11:52:17 (1787586737)
[ 5213.434996] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5220.639867] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 11:52:24 (1787586744)
[ 5220.916244] Lustre: *** cfs_fail_loc=510, val=3***
[ 5229.546083] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 11:52:33 (1787586753)
[ 5230.101217] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5239.673197] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 11:52:43 (1787586763)
[ 5240.007468] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5305.769175] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 11:53:50 (1787586830)
[ 5306.160824] LustreError: 83007:0:(ldlm_flock.c:819:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5308.875160] Lustre: lustre-MDT0000-mdc-ffff93fa49dda000: Connection to lustre-MDT0000 (at 192.168.206.114@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5308.890963] Lustre: Skipped 10 previous similar messages
[ 5308.915215] LustreError: lustre-MDT0000-mdc-ffff93fa49dda000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5308.929610] LustreError: 83021:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff93fa49dda000: inode [0x20000a042:0x8:0x0] mdc close failed: rc = -108
[ 5308.943581] LustreError: 83021:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff93fa49dda000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff93fa46d17f00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5310.223135] LustreError: 83007:0:(ldlm_flock.c:819:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5312.306532] LustreError: 83037:0:(ldlm_flock.c:819:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5315.367505] LustreError: lustre-MDT0000-mdc-ffff93fa49dda000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5315.384784] LustreError: 83051:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff93fa49dda000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff93fa46d17a00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5315.413221] LustreError: 83051:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5316.375406] LustreError: 83037:0:(ldlm_flock.c:819:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5323.277396] LustreError: lustre-MDT0000-mdc-ffff93fa49dda000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5323.680509] LustreError: 83093:0:(ldlm_flock.c:824:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5326.632275] LustreError: lustre-MDT0000-mdc-ffff93fa49dda000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5326.715360] LustreError: 83113:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5326.725093] LustreError: 83113:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5327.751307] LustreError: 83093:0:(ldlm_flock.c:824:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5327.769603] LustreError: 83093:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff93fa49dda000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5327.787499] LustreError: 83093:0:(file.c:251:ll_close_inode_openhandle()) Skipped 1 previous similar message
[ 5327.811592] LustreError: 83120:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5327.828551] LustreError: 83120:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5330.129211] LustreError: 83142:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5330.157199] LustreError: 83142:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5333.748602] LustreError: 83169:0:(ldlm_flock.c:824:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5336.765766] LustreError: lustre-MDT0000-mdc-ffff93fa49dda000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5336.787011] LustreError: 83183:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff93fa49dda000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff93fa46778c00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5336.819379] LustreError: 83183:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5337.815120] LustreError: 83169:0:(ldlm_flock.c:824:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5344.730292] LustreError: lustre-MDT0000-mdc-ffff93fa49dda000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5345.148984] LustreError: 83225:0:(ldlm_flock.c:873:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5347.963626] LustreError: lustre-MDT0000-mdc-ffff93fa49dda000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5347.982879] LustreError: 83240:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff93fa49dda000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff93fa46778100) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5348.018030] LustreError: 83240:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5349.223205] LustreError: 83225:0:(ldlm_flock.c:873:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5352.321843] LustreError: lustre-MDT0000-mdc-ffff93fa49dda000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5352.410177] LustreError: 83273:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5352.429750] LustreError: 83273:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5353.384259] LustreError: 83254:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff93fa49dda000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5361.502732] LustreError: 83325:0:(ldlm_flock.c:819:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5361.509350] LustreError: 83325:0:(ldlm_flock.c:819:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5364.967151] LustreError: lustre-MDT0000-mdc-ffff93fa49dda000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5365.036759] LustreError: 83348:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5365.051388] LustreError: 83348:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5365.591178] LustreError: 83325:0:(ldlm_flock.c:819:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5365.596405] LustreError: 83325:0:(ldlm_flock.c:819:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5365.606817] LustreError: 83325:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff93fa49dda000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5369.065885] Lustre: lustre-MDT0000-mdc-ffff93fa49dda000: Connection restored to 192.168.206.114@tcp (at 192.168.206.114@tcp)
[ 5369.073283] Lustre: Skipped 18 previous similar messages
[ 5375.875777] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 11:54:59 (1787586899)
[ 5394.658179] LustreError: 68261:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5394.673806] LustreError: 68261:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5398.751532] LustreError: 68261:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5398.762531] LustreError: 68261:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5405.367751] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 11:55:29 (1787586929)
[ 5455.816636] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 11:56:19 (1787586979)
[ 5520.795576] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 11:57:25 (1787587045)
[ 5555.042244] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 11:57:59 (1787587079)
[ 5555.633447] Lustre: Mounted lustre-client - version 2.17.57_80_g3abb7b2
[ 5679.547727] Lustre: Unmounted lustre-client
[ 5686.114556] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 12:00:10 (1787587210)
[ 5686.650299] LustreError: 87653:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5689.049470] LustreError: lustre-OST0000-osc-ffff93fa49dda000: operation ost_statfs to node 192.168.206.114@tcp failed: rc = -107
[ 5689.057383] LustreError: Skipped 9 previous similar messages
[ 5689.066926] LustreError: lustre-OST0000-osc-ffff93fa49dda000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5689.084195] LustreError: 87740:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff93fa49dda000: namespace resource [0x240000400:0x422d:0x0].0x0 (ffff93fa46778b00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5689.092693] LustreError: 87740:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5689.295119] LustreError: 87653:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5689.304063] Lustre: 2352:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.206.114@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5695.949800] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 12:00:19 (1787587219)
[ 5753.823153] Lustre: 88320:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787587223/real 1787587223]  req@ffff93fa464c5500 x1874415183749888/t0(0) o101->lustre-MDT0000-mdc-ffff93fa49dda000@192.168.206.114@tcp:12/10 lens 328/344 e 0 to 1 dl 1787587279 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 5753.861377] Lustre: 88320:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
[ 5761.902723] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 12:01:25 (1787587285)
[ 5763.885814] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 5765.844203] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 12:01:29 (1787587289)
[ 5829.217382] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 5830.636805] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 12:02:34 (1787587354)
[ 5889.643546] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 12:03:33 (1787587413)
[ 5891.917770] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 5893.981930] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 12:03:37 (1787587417)
[ 5895.330783] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 5897.407952] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 12:03:41 (1787587421)
[ 5926.066817] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 12:04:09 (1787587449)
[ 5937.684801] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5964.703215] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 192.168.206.114@tcp) was lost; in progress operations using this service will fail
[ 5964.715833] LustreError: Skipped 2 previous similar messages
[ 5964.733345] Lustre: Evicted from MGS (at 192.168.206.114@tcp) after server handle changed from 0x37be1cabc79ac653 to 0x37be1cabc79adff9
[ 5964.736320] Lustre: lustre-MDT0000-mdc-ffff93fa49dda000: Connection to lustre-MDT0000 (at 192.168.206.114@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5964.745782] Lustre: Skipped 2 previous similar messages
[ 5964.774941] Lustre: Skipped 15 previous similar messages
[ 5974.045796] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5975.269513] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5984.292698] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 12:05:08 (1787587508)
[ 5986.573145] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 5988.060501] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 12:05:12 (1787587512)
[ 6000.621705] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 192.168.206.114@tcp) was lost; in progress operations using this service will fail
[ 6000.655846] Lustre: Evicted from MGS (at 192.168.206.114@tcp) after server handle changed from 0x37be1cabc79adff9 to 0x37be1cabc79ae333
[ 6000.680538] Lustre: MGC192.168.206.114@tcp: Connection restored to 192.168.206.114@tcp (at 192.168.206.114@tcp)
[ 6000.685380] Lustre: Skipped 9 previous similar messages
[ 6011.534225] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 12:05:35 (1787587535)
[ 6054.791179] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 12:06:18 (1787587578)
[ 6107.841478] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6113.703678] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6200.303473] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 192.168.206.114@tcp) was lost; in progress operations using this service will fail
[ 6200.311171] LustreError: Skipped 1 previous similar message
[ 6200.322162] Lustre: Evicted from MGS (at 192.168.206.114@tcp) after server handle changed from 0x37be1cabc79ae881 to 0x37be1cabc79af38d
[ 6200.336330] Lustre: Skipped 1 previous similar message
[ 6202.239639] Lustre: 68264:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.114@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6221.539793] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6222.841238] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6258.490528] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6259.982659] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6391.520841] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 12:11:55 (1787587915)
[ 6403.621653] LustreError: lustre-OST0000-osc-ffff93fa49dda000: operation ost_setattr to node 192.168.206.114@tcp failed: rc = -107
[ 6403.646384] LustreError: Skipped 537 previous similar messages
[ 6455.913460] Lustre: DEBUG MARKER: oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6461.257229] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6709.664247] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 12:17:13 (1787588233)
[ 6851.047882] Lustre: lustre-MDT0000-mdc-ffff93fa49dda000: Connection to lustre-MDT0000 (at 192.168.206.114@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6851.056062] Lustre: Skipped 6 previous similar messages
[ 6851.063442] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 192.168.206.114@tcp) was lost; in progress operations using this service will fail
[ 6851.071947] LustreError: Skipped 1 previous similar message
[ 6851.109295] Lustre: Evicted from MGS (at 192.168.206.114@tcp) after server handle changed from 0x37be1cabc79af91a to 0x37be1cabc7aa9b29
[ 6851.116152] Lustre: Skipped 1 previous similar message
[ 6851.124919] Lustre: MGC192.168.206.114@tcp: Connection restored to 192.168.206.114@tcp (at 192.168.206.114@tcp)
[ 6851.135358] Lustre: Skipped 7 previous similar messages
[ 6882.493866] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 12:20:06 (1787588406)
[ 6883.983548] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 6885.751707] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 12:20:09 (1787588409)
[ 6887.821721] LustreError: lustre-MDT0000-mdc-ffff93fa49dda000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6895.170804] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 12:20:19 (1787588419)
[ 7062.066923] LustreError: lustre-OST0000-osc-ffff93fa49dda000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7063.518801] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 12:23:07 (1787588587)
[ 7088.543207] Lustre: 2354:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787588593/real 1787588593]  req@ffff93fa50ce3480 x1874415201513600/t0(0) o4->lustre-OST0000-osc-ffff93fa49dda000@192.168.206.114@tcp:6/4 lens 4584/448 e 0 to 1 dl 1787588613 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7088.568210] Lustre: 2354:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 21 previous similar messages
[ 7105.587821] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 12:23:49 (1787588629)
[ 7107.120508] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7108.593941] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 12:23:52 (1787588632)
[ 7110.076314] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7111.718496] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 12:23:55 (1787588635)
[ 7145.133179] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 12:24:29 (1787588669)
[ 7184.884179] LustreError: MGC192.168.206.114@tcp: Connection to MGS (at 192.168.206.114@tcp) was lost; in progress operations using this service will fail
[ 7184.913714] Lustre: Evicted from MGS (at 192.168.206.114@tcp) after server handle changed from 0x37be1cabc7aa9b29 to 0x37be1cabc7aac3e7
[ 7184.921054] Lustre: Skipped 1 previous similar message
[ 7199.487532] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 12:25:23 (1787588723)
[ 7201.092678] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7202.792556] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 12:25:26 (1787588726)
[ 7204.079932] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7205.754929] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 12:25:29 (1787588729)
[ 7216.072177] Lustre: Unmounted lustre-client
[ 7217.986284] Lustre: Mounted lustre-client - version 2.17.57_80_g3abb7b2
[ 7220.634473] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7238.623111] Lustre: 2353:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787588748/real 1787588748]  req@ffff93fa70c66d80 x1874415201700352/t0(0) o400->MGC192.168.206.114@tcp@192.168.206.114@tcp:26/25 lens 224/224 e 0 to 1 dl 1787588764 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7250.346461] Lustre: 106776:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.114@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 7261.543247] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2