[ 1871.205214] LustreError: lustre-OST0000-osc-ffffa0dcc9d1a800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1871.217759] Lustre: 2362:0:(llite_lib.c:4352:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.111@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1877.281420] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 12:38:48 (1787848728)
[ 1879.115458] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1880.448888] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 12:38:51 (1787848731)
[ 1881.967631] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1883.283810] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 12:38:54 (1787848734)
[ 1886.475812] LustreError: lustre-MDT0000-mdc-ffffa0dcc9d1a800: operation ldlm_enqueue to node 192.168.203.111@tcp failed: rc = -19
[ 1886.490875] LustreError: Skipped 1 previous similar message
[ 1904.160363] LustreError: MGC192.168.203.111@tcp: Connection to MGS (at 192.168.203.111@tcp) was lost; in progress operations using this service will fail
[ 1904.212047] Lustre: Evicted from MGS (at 192.168.203.111@tcp) after server handle changed from 0x50363f7cc907c41a to 0x50363f7cc907da2b
[ 2007.778492] LustreError: lustre-MDT0000-mdc-ffffa0dcc9d1a800: operation ldlm_enqueue to node 192.168.203.111@tcp failed: rc = -19
[ 2007.789104] LustreError: Skipped 1 previous similar message
[ 2025.952284] LustreError: MGC192.168.203.111@tcp: Connection to MGS (at 192.168.203.111@tcp) was lost; in progress operations using this service will fail
[ 2036.144282] Lustre: Evicted from MGS (at 192.168.203.111@tcp) after server handle changed from 0x50363f7cc907da2b to 0x50363f7cc909b4b1
[ 2051.150445] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 12:41:42 (1787848902)
[ 2051.633514] Lustre: *** cfs_fail_loc=305, val=0***
[ 2051.637083] Lustre: Skipped 2 previous similar messages
[ 2091.506064] LustreError: MGC192.168.203.111@tcp: Connection to MGS (at 192.168.203.111@tcp) was lost; in progress operations using this service will fail
[ 2091.532275] Lustre: Evicted from MGS (at 192.168.203.111@tcp) after server handle changed from 0x50363f7cc909b4b1 to 0x50363f7cc909b84d
[ 2106.791675] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2109.214507] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2118.663398] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 12:42:49 (1787848969)
[ 2132.473772] LustreError: MGC192.168.203.111@tcp: Connection to MGS (at 192.168.203.111@tcp) was lost; in progress operations using this service will fail
[ 2132.501393] Lustre: Evicted from MGS (at 192.168.203.111@tcp) after server handle changed from 0x50363f7cc909b84d to 0x50363f7cc909be3c
[ 2132.554501] LustreError: lustre-MDT0000-mdc-ffffa0dcc9d1a800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2153.141757] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 12:43:24 (1787849004)
[ 2158.074578] Lustre: lustre-OST0000-osc-ffffa0dcc9d1a800: Connection to lustre-OST0000 (at 192.168.203.111@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2158.088144] Lustre: Skipped 13 previous similar messages
[ 2165.384779] LustreError: lustre-OST0000-osc-ffffa0dcc9d1a800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2165.414498] Lustre: lustre-OST0000-osc-ffffa0dcc9d1a800: Connection restored to 192.168.203.111@tcp (at 192.168.203.111@tcp)
[ 2165.422791] Lustre: Skipped 16 previous similar messages
[ 2182.942231] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 12:43:53 (1787849033)
[ 2196.134507] LustreError: lustre-MDT0000-mdc-ffffa0dcc9d1a800: operation ldlm_enqueue to node 192.168.203.111@tcp failed: rc = -19
[ 2215.394942] LustreError: MGC192.168.203.111@tcp: Connection to MGS (at 192.168.203.111@tcp) was lost; in progress operations using this service will fail
[ 2215.430863] Lustre: Evicted from MGS (at 192.168.203.111@tcp) after server handle changed from 0x50363f7cc909be3c to 0x50363f7cc90a15f4
[ 2241.491372] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2243.330269] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2325.984704] Lustre: 2362:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787849162/real 1787849162]  req@ffffa0dcc4908b40 x1874693403222272/t0(0) o400->MGC192.168.203.111@tcp@192.168.203.111@tcp:26/25 lens 224/224 e 0 to 1 dl 1787849178 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2326.010583] Lustre: 2362:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 13 previous similar messages
[ 2326.023496] LustreError: MGC192.168.203.111@tcp: Connection to MGS (at 192.168.203.111@tcp) was lost; in progress operations using this service will fail
[ 2326.048631] Lustre: Evicted from MGS (at 192.168.203.111@tcp) after server handle changed from 0x50363f7cc90a15f4 to 0x50363f7cc90c02d3
[ 2352.760703] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2354.952689] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2419.556078] LustreError: lustre-MDT0000-mdc-ffffa0dcc9d1a800: operation mds_reint to node 192.168.203.111@tcp failed: rc = -19
[ 2419.566980] LustreError: Skipped 1 previous similar message
[ 2439.456588] LustreError: MGC192.168.203.111@tcp: Connection to MGS (at 192.168.203.111@tcp) was lost; in progress operations using this service will fail
[ 2439.516614] Lustre: Evicted from MGS (at 192.168.203.111@tcp) after server handle changed from 0x50363f7cc90c02d3 to 0x50363f7cc90dd572
[ 2463.458876] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2465.286707] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2493.875776] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 12:49:04 (1787849344)
[ 2518.505828] LustreError: MGC192.168.203.111@tcp: Connection to MGS (at 192.168.203.111@tcp) was lost; in progress operations using this service will fail
[ 2518.527132] Lustre: Evicted from MGS (at 192.168.203.111@tcp) after server handle changed from 0x50363f7cc90dd572 to 0x50363f7cc90eab54
[ 2525.035268] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2556.960336] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2593.500962] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2628.529210] Lustre: 15944:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.111@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2642.502600] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2687.008240] LustreError: MGC192.168.203.111@tcp: Connection to MGS (at 192.168.203.111@tcp) was lost; in progress operations using this service will fail
[ 2687.032499] LustreError: Skipped 3 previous similar messages
[ 2697.205259] Lustre: Evicted from MGS (at 192.168.203.111@tcp) after server handle changed from 0x50363f7cc90f84fc to 0x50363f7cc90fd7c8
[ 2697.232472] Lustre: Skipped 3 previous similar messages
[ 2698.018669] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2725.734345] LustreError: lustre-MDT0000-mdc-ffffa0dcc9d1a800: operation mds_reint to node 192.168.203.111@tcp failed: rc = -19
[ 2725.744724] LustreError: Skipped 9 previous similar messages
[ 2751.114279] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2783.750575] Lustre: lustre-MDT0000-mdc-ffffa0dcc9d1a800: Connection to lustre-MDT0000 (at 192.168.203.111@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2783.768084] Lustre: Skipped 9 previous similar messages
[ 2803.722542] Lustre: MGC192.168.203.111@tcp: Connection restored to 192.168.203.111@tcp (at 192.168.203.111@tcp)
[ 2803.729308] Lustre: Skipped 18 previous similar messages
[ 2804.786736] Lustre: 15944:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.111@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2844.117430] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 12:54:55 (1787849695)
[ 2890.639401] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 2892.286308] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3231.696199] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3233.897375] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3519.300844] LustreError: lustre-OST0000-osc-ffffa0dcc9d1a800: operation ldlm_enqueue to node 192.168.203.111@tcp failed: rc = -107
[ 3519.322256] LustreError: Skipped 3 previous similar messages
[ 3519.324300] Lustre: lustre-OST0000-osc-ffffa0dcc9d1a800: Connection to lustre-OST0000 (at 192.168.203.111@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3519.337014] Lustre: Skipped 2 previous similar messages
[ 3539.966453] Lustre: lustre-OST0000-osc-ffffa0dcc9d1a800: Connection restored to 192.168.203.111@tcp (at 192.168.203.111@tcp)
[ 3539.974484] Lustre: Skipped 3 previous similar messages
[ 3554.585961] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3556.096085] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3815.882246] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 13:11:06 (1787850666)
[ 3834.336278] Lustre: 50584:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787850670/real 1787850670]  req@ffffa0dcc49a52c0 x1874693426687872/t0(0) o101->lustre-MDT0000-mdc-ffffa0dcc9d1a800@192.168.203.111@tcp:12/10 lens 576/1152 e 0 to 1 dl 1787850686 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3834.367334] Lustre: 50584:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
[ 3842.839866] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 13:11:33 (1787850693)
[ 3869.763305] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 13:12:00 (1787850720)
[ 3894.509964] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 13:12:25 (1787850745)
[ 3895.169330] Lustre: Mounted lustre-client - version 2.17.57_87_ge30c52f
[ 3925.984618] Lustre: 2361:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787850762/real 1787850762]  req@ffffa0dcc49a52c0 x1874693426706944/t0(0) o400->lustre-MDT0000-mdc-ffffa0dcc9d1a800@192.168.203.111@tcp:12/10 lens 224/224 e 0 to 1 dl 1787850778 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3926.024780] Lustre: 2361:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 3926.038334] LustreError: MGC192.168.203.111@tcp: Connection to MGS (at 192.168.203.111@tcp) was lost; in progress operations using this service will fail
[ 3926.060370] LustreError: Skipped 2 previous similar messages
[ 3936.230445] Lustre: Evicted from MGS (at 192.168.203.111@tcp) after server handle changed from 0x50363f7cc910d901 to 0x50363f7cc9279247
[ 3936.246978] Lustre: Skipped 2 previous similar messages
[ 3941.722604] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 3943.528772] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3945.459620] Lustre: Unmounted lustre-client
[ 3951.492977] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 13:13:22 (1787850802)
[ 4088.800246] Lustre: 2361:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787850924/real 1787850924]  req@ffffa0dcc49a6580 x1874693426740480/t0(0) o4->lustre-OST0000-osc-ffffa0dcc9d1a800@192.168.203.111@tcp:6/4 lens 488/448 e 0 to 1 dl 1787850940 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4088.827731] Lustre: 2361:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 62 previous similar messages
[ 4121.568252] Lustre: lustre-OST0000-osc-ffffa0dcc9d1a800: Connection to lustre-OST0000 (at 192.168.203.111@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4121.583591] Lustre: Skipped 14 previous similar messages
[ 4153.324865] Lustre: lustre-OST0000-osc-ffffa0dcc9d1a800: Connection restored to 192.168.203.111@tcp (at 192.168.203.111@tcp)
[ 4153.345522] Lustre: Skipped 17 previous similar messages
[ 4246.103196] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 13:18:16 (1787851096)
[ 4295.366344] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 13:19:06 (1787851146)
[ 4298.685639] Lustre: Unmounted lustre-client
[ 4322.811572] Lustre: Mounted lustre-client - version 2.17.57_87_ge30c52f
[ 4329.316318] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 13:19:40 (1787851180)
[ 4329.693236] LustreError: 56517:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4330.745070] LustreError: 56517:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4330.928337] Lustre: *** cfs_fail_loc=305, val=0***
[ 4354.587914] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 13:20:05 (1787851205)
[ 4355.131722] Lustre: Mounted lustre-client - version 2.17.57_87_ge30c52f
[ 4357.592596] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4367.910105] Lustre: Unmounted lustre-client
[ 4374.452385] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 13:20:25 (1787851225)
[ 4463.308610] LustreError: lustre-MDT0000-mdc-ffffa0dcc42a7800: operation mds_reint to node 192.168.203.111@tcp failed: rc = -19
[ 4484.064236] Lustre: 2361:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787851320/real 1787851320]  req@ffffa0dcc36b9680 x1874693429413504/t0(0) o400->MGC192.168.203.111@tcp@192.168.203.111@tcp:26/25 lens 224/224 e 0 to 1 dl 1787851336 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4484.109697] Lustre: 2361:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 76 previous similar messages
[ 4484.133041] LustreError: MGC192.168.203.111@tcp: Connection to MGS (at 192.168.203.111@tcp) was lost; in progress operations using this service will fail
[ 4494.315083] Lustre: Evicted from MGS (at 192.168.203.111@tcp) after server handle changed from 0x50363f7cc927ad1a to 0x50363f7cc92a5ef5
[ 4496.171280] Lustre: 55915:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.111@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 4654.152641] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 13:25:05 (1787851505)
[ 4658.705436] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4675.559199] LustreError: MGC192.168.203.111@tcp: Connection to MGS (at 192.168.203.111@tcp) was lost; in progress operations using this service will fail
[ 4675.608442] Lustre: Evicted from MGS (at 192.168.203.111@tcp) after server handle changed from 0x50363f7cc92a5ef5 to 0x50363f7cc92f2deb
[ 4675.661748] LustreError: lustre-MDT0000-mdc-ffffa0dcc42a7800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4687.324600] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 13:25:38 (1787851538)
[ 4687.958719] Lustre: Mounted lustre-client - version 2.17.57_87_ge30c52f
[ 4695.352839] LustreError: lustre-OST0000-osc-ffffa0dcc42a7800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4707.551802] Lustre: Unmounted lustre-client
[ 4714.777408] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 13:26:05 (1787851565)
[ 4721.218446] LustreError: lustre-MDT0000-mdc-ffffa0dcc42a7800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4721.236558] LustreError: 60910:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4728.905474] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 13:26:19 (1787851579)
[ 4729.079284] Lustre: lustre-MDT0000-mdc-ffffa0dcc42a7800: Connection to lustre-MDT0000 (at 192.168.203.111@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4729.109215] Lustre: Skipped 16 previous similar messages
[ 4729.113744] LustreError: 61609:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4734.432163] LustreError: 61609:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=0
[ 4734.455411] LustreError: 61609:0:(import.c:711:ptlrpc_connect_import_locked()) already connected
[ 4735.103248] LustreError: lustre-MDT0000-mdc-ffffa0dcc42a7800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4735.126301] LustreError: 61630:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4735.182885] LustreError: 61635:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4736.264068] LustreError: 61642:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4736.283454] LustreError: 61642:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4738.615616] LustreError: 61664:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4738.637443] LustreError: 61664:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4743.154190] LustreError: 61709:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4743.177452] LustreError: 61709:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4751.646119] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 13:26:42 (1787851602)
[ 4793.333806] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4795.070482] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4804.695979] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 13:27:35 (1787851655)
[ 4829.121613] Lustre: lustre-OST0000-osc-ffffa0dcc42a7800: Connection restored to 192.168.203.111@tcp (at 192.168.203.111@tcp)
[ 4829.137518] Lustre: Skipped 17 previous similar messages
[ 4844.429566] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4845.714558] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4857.526710] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 13:28:27 (1787851707)
[ 4924.275873] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4925.905799] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4934.360472] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 13:29:45 (1787851785)
[ 4972.779224] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4974.268537] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4979.719532] Lustre: Unmounted lustre-client
[ 5030.591489] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5033.194287] Lustre: Mounted lustre-client - version 2.17.57_87_ge30c52f
[ 5040.221341] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 13:31:31 (1787851891)
[ 5042.600062] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5044.324295] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 13:31:35 (1787851895)
[ 5079.554785] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 13:32:10 (1787851930)
[ 5081.418753] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5083.672533] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 13:32:13 (1787851933)
[ 5085.496570] Lustre: *** cfs_fail_loc=805, val=0***
[ 5085.575837] Lustre: Mounted lustre-client - version 2.17.57_87_ge30c52f
[ 5091.551618] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5112.289452] Lustre: 2360:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787851948/real 1787851948]  req@ffffa0dcc5c4d2c0 x1874693431473280/t0(0) o400->lustre-MDT0000-mdc-ffffa0dccbbc7800@192.168.203.111@tcp:12/10 lens 224/224 e 0 to 1 dl 1787851964 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5112.337642] Lustre: 2360:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5112.346633] LustreError: MGC192.168.203.111@tcp: Connection to MGS (at 192.168.203.111@tcp) was lost; in progress operations using this service will fail
[ 5122.546929] Lustre: Evicted from MGS (at 192.168.203.111@tcp) after server handle changed from 0x50363f7cc92f37d8 to 0x50363f7cc92f3b74
[ 5128.629787] Lustre: Unmounted lustre-client
[ 5136.200720] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 13:33:06 (1787851986)
[ 5163.565232] LustreError: 2358:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffffa0dcc7561680 x1874693431484160/t94489280516(94489280516) o101->lustre-MDT0000-mdc-ffffa0dccbbc7800@192.168.203.111@tcp:12/10 lens 664/608 e 0 to 0 dl 1787852031 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5176.201346] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5177.986933] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5187.370736] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 13:33:58 (1787852038)
[ 5188.732551] LustreError: lustre-OST0000-osc-ffffa0dccbbc7800: operation ost_write to node 192.168.203.111@tcp failed: rc = -107
[ 5188.745022] LustreError: Skipped 3 previous similar messages
[ 5188.772688] LustreError: lustre-OST0000-osc-ffffa0dccbbc7800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5188.788391] Lustre: 2362:0:(llite_lib.c:4352:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.111@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5188.860899] LustreError: 72819:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffffa0dccbbc7800: namespace resource [0x240000400:0x41e2:0x0].0x0 (ffffa0dcc5fd4800) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5197.176906] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 13:34:07 (1787852047)
[ 5198.525989] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5199.955180] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 13:34:11 (1787852051)
[ 5201.716967] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5203.282106] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 13:34:14 (1787852054)
[ 5205.069947] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5206.832693] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 13:34:17 (1787852057)
[ 5208.366427] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5210.032277] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 13:34:20 (1787852060)
[ 5211.622302] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5213.327422] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 13:34:24 (1787852064)
[ 5214.598163] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5216.289158] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 13:34:27 (1787852067)
[ 5217.911192] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5219.724192] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 13:34:30 (1787852070)
[ 5221.187696] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5223.192543] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 13:34:33 (1787852073)
[ 5224.918353] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5227.507867] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 13:34:37 (1787852077)
[ 5229.964447] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5232.962940] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 13:34:42 (1787852082)
[ 5235.391526] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5237.978166] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 13:34:48 (1787852088)
[ 5241.281522] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5243.096562] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 13:34:53 (1787852093)
[ 5276.428356] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 13:35:26 (1787852126)
[ 5311.171399] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 13:36:01 (1787852161)
[ 5311.878390] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5322.639933] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 13:36:13 (1787852173)
[ 5323.046043] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5333.436418] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 13:36:24 (1787852184)
[ 5333.745387] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5341.283279] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 13:36:32 (1787852192)
[ 5341.677837] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5352.156272] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 13:36:42 (1787852202)
[ 5352.484910] Lustre: *** cfs_fail_loc=510, val=3***
[ 5362.259403] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 13:36:52 (1787852212)
[ 5362.826748] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5422.560694] Lustre: lustre-OST0000-osc-ffffa0dccbbc7800: Connection to lustre-OST0000 (at 192.168.203.111@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5422.571112] Lustre: Skipped 10 previous similar messages
[ 5430.991044] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 13:38:01 (1787852281)
[ 5431.624463] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5497.565536] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 13:39:08 (1787852348)
[ 5497.889584] LustreError: 83039:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5500.849872] LustreError: lustre-MDT0000-mdc-ffffa0dccbbc7800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5500.867112] LustreError: 83053:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffffa0dccbbc7800: namespace resource [0x20000a042:0x11:0x0].0xc (ffffa0dcc7739400) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5500.915800] Lustre: lustre-MDT0000-mdc-ffffa0dccbbc7800: Connection restored to 192.168.203.111@tcp (at 192.168.203.111@tcp)
[ 5500.932244] Lustre: Skipped 9 previous similar messages
[ 5501.984122] LustreError: 83039:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5504.125627] LustreError: 83069:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5507.067905] LustreError: lustre-MDT0000-mdc-ffffa0dccbbc7800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5507.094440] LustreError: 83083:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffffa0dccbbc7800: namespace resource [0x20000a042:0x11:0x0].0xc (ffffa0dcc5dfcd00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5507.122995] LustreError: 83083:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5508.200227] LustreError: 83069:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5515.166243] LustreError: lustre-MDT0000-mdc-ffffa0dccbbc7800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5515.568147] LustreError: 83125:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5518.525241] LustreError: lustre-MDT0000-mdc-ffffa0dccbbc7800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5518.621777] LustreError: 83145:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5518.635108] LustreError: 83145:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5519.665386] LustreError: 83125:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5519.674171] LustreError: 83125:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffffa0dccbbc7800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5519.730588] LustreError: 83151:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5519.738065] LustreError: 83151:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5522.123084] LustreError: 83173:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5522.138295] LustreError: 83173:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5525.725282] LustreError: 83200:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5528.575241] LustreError: lustre-MDT0000-mdc-ffffa0dccbbc7800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5528.600979] LustreError: 83215:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffffa0dccbbc7800: namespace resource [0x20000a042:0x11:0x0].0xc (ffffa0dcc7739500) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5528.637959] LustreError: 83215:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5529.808143] LustreError: 83200:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5536.710526] LustreError: lustre-MDT0000-mdc-ffffa0dccbbc7800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5537.216535] LustreError: 83256:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5540.201146] LustreError: lustre-MDT0000-mdc-ffffa0dccbbc7800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5540.223952] LustreError: 83271:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffffa0dccbbc7800: namespace resource [0x20000a042:0x11:0x0].0xc (ffffa0dcc5fd4100) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5540.272643] LustreError: 83271:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5541.280973] LustreError: 83256:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5544.253937] LustreError: lustre-MDT0000-mdc-ffffa0dccbbc7800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5544.290681] LustreError: 83304:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5544.294842] LustreError: 83304:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5545.467891] LustreError: 83284:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffffa0dccbbc7800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5553.562289] LustreError: 83356:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5553.598112] LustreError: 83356:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5557.325967] LustreError: lustre-MDT0000-mdc-ffffa0dccbbc7800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5557.452501] LustreError: 83378:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5557.467639] LustreError: 83378:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5557.600176] LustreError: 83356:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5557.608155] LustreError: 83356:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5557.619599] LustreError: 83356:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffffa0dccbbc7800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5570.593360] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 13:40:21 (1787852421)
[ 5590.723258] LustreError: 79783:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5590.730682] LustreError: 79783:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5594.736253] LustreError: 79783:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5594.754687] LustreError: 79783:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5603.738307] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 13:40:53 (1787852453)
[ 5655.280473] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 13:41:45 (1787852505)
[ 5712.864679] Lustre: 85535:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787852509/real 1787852509]  req@ffffa0dcc76d2d00 x1874693431613952/t0(0) o101->lustre-MDT0000-mdc-ffffa0dccbbc7800@192.168.203.111@tcp:12/10 lens 576/1152 e 0 to 1 dl 1787852565 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'stat.0' uid:0 gid:0 projid:0
[ 5712.887787] Lustre: 85535:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 21 previous similar messages
[ 5719.193697] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 13:42:49 (1787852569)
[ 5755.834924] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 13:43:26 (1787852606)
[ 5756.414905] Lustre: Mounted lustre-client - version 2.17.57_87_ge30c52f
[ 5881.709248] Lustre: Unmounted lustre-client
[ 5890.113735] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 13:45:40 (1787852740)
[ 5891.115415] LustreError: 87686:0:(osc_request.c:3192:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5893.943911] LustreError: lustre-OST0000-osc-ffffa0dccbbc7800: operation ldlm_enqueue to node 192.168.203.111@tcp failed: rc = -107
[ 5893.951736] LustreError: Skipped 9 previous similar messages
[ 5893.967223] LustreError: lustre-OST0000-osc-ffffa0dccbbc7800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5893.996663] LustreError: 87768:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffffa0dccbbc7800: namespace resource [0x240000400:0x420d:0x0].0x0 (ffffa0dcc7739100) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5894.037372] LustreError: 87768:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5894.400107] LustreError: 87686:0:(osc_request.c:3192:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5894.409503] Lustre: 2359:0:(llite_lib.c:4352:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.111@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5902.340122] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 13:45:52 (1787852752)
[ 5968.856523] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 13:46:59 (1787852819)
[ 5970.590660] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 5972.740184] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 13:47:03 (1787852823)
[ 6030.304301] Lustre: lustre-MDT0000-mdc-ffffa0dccbbc7800: Connection to lustre-MDT0000 (at 192.168.203.111@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6030.348716] Lustre: Skipped 15 previous similar messages
[ 6042.869308] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6045.597904] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 13:48:15 (1787852895)
[ 6105.071066] Lustre: lustre-MDT0000-mdc-ffffa0dccbbc7800: Connection restored to 192.168.203.111@tcp (at 192.168.203.111@tcp)
[ 6105.086571] Lustre: Skipped 13 previous similar messages
[ 6107.388553] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 13:49:17 (1787852957)
[ 6110.180727] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6112.127917] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 13:49:22 (1787852962)
[ 6113.403712] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6115.086186] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 13:49:25 (1787852965)
[ 6148.432731] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 13:49:59 (1787852999)
[ 6158.490332] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6181.856325] LustreError: MGC192.168.203.111@tcp: Connection to MGS (at 192.168.203.111@tcp) was lost; in progress operations using this service will fail
[ 6181.876172] LustreError: Skipped 2 previous similar messages
[ 6192.103515] Lustre: Evicted from MGS (at 192.168.203.111@tcp) after server handle changed from 0x50363f7cc92f463a to 0x50363f7cc92f5e3c
[ 6192.112474] Lustre: Skipped 2 previous similar messages
[ 6206.821853] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6208.220555] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6219.160140] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 13:51:09 (1787853069)
[ 6221.709316] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6222.768692] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 13:51:14 (1787853074)
[ 6238.182809] LustreError: MGC192.168.203.111@tcp: Connection to MGS (at 192.168.203.111@tcp) was lost; in progress operations using this service will fail
[ 6238.207381] Lustre: Evicted from MGS (at 192.168.203.111@tcp) after server handle changed from 0x50363f7cc92f5e3c to 0x50363f7cc92f62f7
[ 6248.573752] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 13:51:39 (1787853099)
[ 6292.586128] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 13:52:23 (1787853143)
[ 6312.928952] Lustre: 2360:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787853110/real 1787853110]  req@ffffa0dcc5c4e1c0 x1874693431712384/t0(0) o400->lustre-MDT0000-mdc-ffffa0dccbbc7800@192.168.203.111@tcp:12/10 lens 224/224 e 0 to 1 dl 1787853165 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6312.979617] Lustre: 2360:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 14 previous similar messages
[ 6345.251436] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6348.697752] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6433.761174] LustreError: MGC192.168.203.111@tcp: Connection to MGS (at 192.168.203.111@tcp) was lost; in progress operations using this service will fail
[ 6433.776420] LustreError: Skipped 1 previous similar message
[ 6442.984923] Lustre: Evicted from MGS (at 192.168.203.111@tcp) after server handle changed from 0x50363f7cc92f667e to 0x50363f7cc92f744d
[ 6442.997260] Lustre: Skipped 1 previous similar message
[ 6455.883971] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6457.592125] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6496.013464] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6497.444988] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6622.035504] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 13:57:52 (1787853472)
[ 6633.238767] LustreError: lustre-OST0000-osc-ffffa0dccbbc7800: operation ost_setattr to node 192.168.203.111@tcp failed: rc = -19
[ 6633.241458] Lustre: lustre-OST0000-osc-ffffa0dccbbc7800: Connection to lustre-OST0000 (at 192.168.203.111@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6633.247445] LustreError: Skipped 209 previous similar messages
[ 6633.256769] Lustre: Skipped 7 previous similar messages
[ 6682.660212] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6685.321651] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6936.367429] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 14:03:07 (1787853787)
[ 7077.860933] LustreError: MGC192.168.203.111@tcp: Connection to MGS (at 192.168.203.111@tcp) was lost; in progress operations using this service will fail
[ 7077.876113] LustreError: Skipped 1 previous similar message
[ 7077.896464] Lustre: Evicted from MGS (at 192.168.203.111@tcp) after server handle changed from 0x50363f7cc92f7859 to 0x50363f7cc93f1b56
[ 7077.904808] Lustre: Skipped 1 previous similar message
[ 7077.913637] Lustre: MGC192.168.203.111@tcp: Connection restored to 192.168.203.111@tcp (at 192.168.203.111@tcp)
[ 7077.928809] Lustre: Skipped 10 previous similar messages
[ 7104.666903] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 14:05:55 (1787853955)
[ 7106.112661] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7107.820673] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 14:05:58 (1787853958)
[ 7109.677844] LustreError: lustre-MDT0000-mdc-ffffa0dccbbc7800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7116.147710] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 14:06:07 (1787853967)
[ 7283.120267] LustreError: lustre-OST0000-osc-ffffa0dccbbc7800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7285.310991] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 14:08:55 (1787854135)
[ 7310.432388] Lustre: 2360:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787854142/real 1787854142]  req@ffffa0dcc71743c0 x1874693449389568/t0(0) o4->lustre-OST0000-osc-ffffa0dccbbc7800@192.168.203.111@tcp:6/4 lens 4584/448 e 0 to 1 dl 1787854162 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7310.462484] Lustre: 2360:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 10 previous similar messages
[ 7310.474377] Lustre: lustre-OST0000-osc-ffffa0dccbbc7800: Connection to lustre-OST0000 (at 192.168.203.111@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7310.510401] Lustre: Skipped 3 previous similar messages
[ 7326.186971] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 14:09:37 (1787854177)
[ 7327.319707] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7328.651826] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 14:09:39 (1787854179)
[ 7329.920045] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7331.242352] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 14:09:42 (1787854182)
[ 7363.730952] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 14:10:14 (1787854214)
[ 7400.942678] LustreError: MGC192.168.203.111@tcp: Connection to MGS (at 192.168.203.111@tcp) was lost; in progress operations using this service will fail
[ 7400.969865] Lustre: Evicted from MGS (at 192.168.203.111@tcp) after server handle changed from 0x50363f7cc93f1b56 to 0x50363f7cc93f463d
[ 7400.980573] Lustre: Skipped 1 previous similar message
[ 7415.079736] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 14:11:05 (1787854265)
[ 7416.462029] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7417.623528] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 14:11:08 (1787854268)
[ 7418.850683] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7420.066927] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 14:11:11 (1787854271)
[ 7428.983390] Lustre: Unmounted lustre-client
[ 7430.352960] Lustre: Mounted lustre-client - version 2.17.57_87_ge30c52f
[ 7432.780849] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7469.252552] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2