[ 1884.903204] LustreError: 34761:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 1887.206822] LustreError: 487:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 1887.228926] LustreError: lustre-OST0000-osc-ffff8c3b52451000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1895.000098] LustreError: 34761:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 awake
[ 1895.005905] Lustre: 2359:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.206.128@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1895.137077] LustreError: 487:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout interrupted
[ 1895.148737] Lustre: 2356:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.206.128@tcp:/lustre/fid: [0x200000404:0x36:0x0]// may get corrupted (rc -108)
[ 1896.951378] Lustre: DEBUG MARKER: recovery-small test_24b: @@@@@@ IGNORE (bz5494): multiop didn't fail fsync: 108 or close: 0
[ 1904.936456] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 00:15:33 (1790741733)
[ 1906.585891] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1907.890865] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 00:15:36 (1790741736)
[ 1909.454402] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1911.223971] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 00:15:39 (1790741739)
[ 1914.489284] LustreError: lustre-MDT0000-mdc-ffff8c3b52451000: operation mds_reint to node 192.168.206.128@tcp failed: rc = -19
[ 1933.280426] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 192.168.206.128@tcp) was lost; in progress operations using this service will fail
[ 1933.318050] Lustre: Evicted from MGS (at 192.168.206.128@tcp) after server handle changed from 0xcc7f1be12857cbac to 0xcc7f1be12857e60f
[ 2033.536682] LustreError: lustre-MDT0000-mdc-ffff8c3b52451000: operation mds_reint to node 192.168.206.128@tcp failed: rc = -19
[ 2033.543710] LustreError: Skipped 1 previous similar message
[ 2050.592372] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 192.168.206.128@tcp) was lost; in progress operations using this service will fail
[ 2060.797993] Lustre: Evicted from MGS (at 192.168.206.128@tcp) after server handle changed from 0xcc7f1be12857e60f to 0xcc7f1be1285994ea
[ 2078.312482] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 00:18:26 (1790741906)
[ 2078.849930] Lustre: *** cfs_fail_loc=305, val=0***
[ 2078.855323] Lustre: Skipped 4 previous similar messages
[ 2098.562622] LustreError: lustre-OST0001-osc-ffff8c3b52451000: operation ost_connect to node 192.168.206.128@tcp failed: rc = -75
[ 2124.256422] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 192.168.206.128@tcp) was lost; in progress operations using this service will fail
[ 2134.503330] Lustre: Evicted from MGS (at 192.168.206.128@tcp) after server handle changed from 0xcc7f1be1285994ea to 0xcc7f1be12859986a
[ 2151.124563] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2152.601793] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2161.649842] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 00:19:50 (1790741990)
[ 2175.468391] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 192.168.206.128@tcp) was lost; in progress operations using this service will fail
[ 2175.517238] Lustre: Evicted from MGS (at 192.168.206.128@tcp) after server handle changed from 0xcc7f1be12859986a to 0xcc7f1be128599d10
[ 2175.638450] LustreError: lustre-MDT0000-mdc-ffff8c3b52451000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2199.425495] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 00:20:27 (1790742027)
[ 2212.039573] LustreError: lustre-OST0000-osc-ffff8c3b52451000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2232.967839] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 00:21:01 (1790742061)
[ 2246.385403] LustreError: lustre-MDT0000-mdc-ffff8c3b52451000: operation mds_reint to node 192.168.206.128@tcp failed: rc = -19
[ 2263.520668] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 192.168.206.128@tcp) was lost; in progress operations using this service will fail
[ 2273.774426] Lustre: Evicted from MGS (at 192.168.206.128@tcp) after server handle changed from 0xcc7f1be128599d10 to 0xcc7f1be12859e07e
[ 2291.268099] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2294.151850] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2379.232679] Lustre: 2358:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790742193/real 1790742193]  req@ffff8c3b46351500 x1877726892331392/t0(0) o400->MGC192.168.206.128@tcp@192.168.206.128@tcp:26/25 lens 224/224 e 0 to 1 dl 1790742209 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2379.262311] Lustre: 2358:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 12 previous similar messages
[ 2379.274663] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 192.168.206.128@tcp) was lost; in progress operations using this service will fail
[ 2389.488243] Lustre: Evicted from MGS (at 192.168.206.128@tcp) after server handle changed from 0xcc7f1be12859e07e to 0xcc7f1be1285b9fe4
[ 2408.328560] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2410.839591] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2495.984280] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 192.168.206.128@tcp) was lost; in progress operations using this service will fail
[ 2496.015571] Lustre: Evicted from MGS (at 192.168.206.128@tcp) after server handle changed from 0xcc7f1be1285b9fe4 to 0xcc7f1be1285d20ba
[ 2496.037269] Lustre: MGC192.168.206.128@tcp: Connection restored to 192.168.206.128@tcp (at 192.168.206.128@tcp)
[ 2496.046868] Lustre: Skipped 14 previous similar messages
[ 2497.976887] Lustre: 15900:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.128@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2523.226457] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2525.075263] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2554.438736] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 00:26:23 (1790742383)
[ 2558.917276] LustreError: lustre-MDT0000-mdc-ffff8c3b52451000: operation ldlm_enqueue to node 192.168.206.128@tcp failed: rc = -19
[ 2558.923234] Lustre: lustre-MDT0000-mdc-ffff8c3b52451000: Connection to lustre-MDT0000 (at 192.168.206.128@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2558.929852] LustreError: Skipped 4 previous similar messages
[ 2558.948871] Lustre: Skipped 9 previous similar messages
[ 2577.952900] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 192.168.206.128@tcp) was lost; in progress operations using this service will fail
[ 2587.123485] Lustre: Evicted from MGS (at 192.168.206.128@tcp) after server handle changed from 0xcc7f1be1285d20ba to 0xcc7f1be1285ddc0f
[ 2595.010348] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2626.051306] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2654.963570] Lustre: 15900:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.128@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2671.725863] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2711.756876] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2754.017090] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 192.168.206.128@tcp) was lost; in progress operations using this service will fail
[ 2754.024036] LustreError: Skipped 3 previous similar messages
[ 2764.212491] Lustre: Evicted from MGS (at 192.168.206.128@tcp) after server handle changed from 0xcc7f1be1285e8c6d to 0xcc7f1be1285edcdf
[ 2764.235455] Lustre: Skipped 3 previous similar messages
[ 2765.632158] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2816.966318] Lustre: 15900:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.128@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2832.873729] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2919.253575] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 00:32:27 (1790742747)
[ 2977.154408] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 2979.698917] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3265.090619] LustreError: lustre-OST0000-osc-ffff8c3b52451000: operation ldlm_enqueue to node 192.168.206.128@tcp failed: rc = -19
[ 3265.100428] LustreError: Skipped 10 previous similar messages
[ 3265.107679] Lustre: lustre-OST0000-osc-ffff8c3b52451000: Connection to lustre-OST0000 (at 192.168.206.128@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3265.120821] Lustre: Skipped 7 previous similar messages
[ 3286.841784] Lustre: lustre-OST0000-osc-ffff8c3b52451000: Connection restored to 192.168.206.128@tcp (at 192.168.206.128@tcp)
[ 3286.856314] Lustre: Skipped 15 previous similar messages
[ 3306.171880] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3308.483665] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3641.082859] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3643.228292] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3892.653145] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 00:48:41 (1790743721)
[ 3910.624172] Lustre: 50842:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790743724/real 1790743724]  req@ffff8c3b46488700 x1877726912953216/t0(0) o101->lustre-MDT0000-mdc-ffff8c3b52451000@192.168.206.128@tcp:12/10 lens 576/1152 e 0 to 1 dl 1790743740 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3910.660569] Lustre: 50842:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
[ 3910.677495] Lustre: lustre-MDT0000-mdc-ffff8c3b52451000: Connection to lustre-MDT0000 (at 192.168.206.128@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3910.700723] Lustre: Skipped 1 previous similar message
[ 3910.781641] Lustre: lustre-MDT0000-mdc-ffff8c3b52451000: Connection restored to 192.168.206.128@tcp (at 192.168.206.128@tcp)
[ 3910.794551] Lustre: Skipped 1 previous similar message
[ 3918.958874] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 00:49:07 (1790743747)
[ 3945.419623] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 00:49:34 (1790743774)
[ 3971.136326] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 00:49:59 (1790743799)
[ 3971.685733] Lustre: Mounted lustre-client - version 2.17.59_3_g21e240e
[ 4002.019612] Lustre: 2356:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790743816/real 1790743816]  req@ffff8c3b464a5c00 x1877726912971904/t0(0) o400->MGC192.168.206.128@tcp@192.168.206.128@tcp:26/25 lens 224/224 e 0 to 1 dl 1790743832 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4002.074874] Lustre: 2356:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 4002.083973] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 192.168.206.128@tcp) was lost; in progress operations using this service will fail
[ 4002.104254] LustreError: Skipped 2 previous similar messages
[ 4012.339568] Lustre: Evicted from MGS (at 192.168.206.128@tcp) after server handle changed from 0xcc7f1be1285ff7fd to 0xcc7f1be12873cfaa
[ 4012.356948] Lustre: Skipped 2 previous similar messages
[ 4030.063281] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4031.999522] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4034.267299] Lustre: Unmounted lustre-client
[ 4041.280406] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 00:51:09 (1790743869)
[ 4163.040317] Lustre: 2359:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790743977/real 1790743977]  req@ffff8c3b462cc380 x1877726913007104/t0(0) o4->lustre-OST0000-osc-ffff8c3b52451000@192.168.206.128@tcp:6/4 lens 488/448 e 0 to 1 dl 1790743993 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4163.075582] Lustre: 2359:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 52 previous similar messages
[ 4338.014957] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 00:56:06 (1790744166)
[ 4386.930360] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 00:56:55 (1790744215)
[ 4390.735809] Lustre: Unmounted lustre-client
[ 4412.081488] Lustre: Mounted lustre-client - version 2.17.59_3_g21e240e
[ 4419.691551] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 00:57:27 (1790744247)
[ 4420.025563] LustreError: 56771:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4421.074601] LustreError: 56771:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4421.375753] Lustre: *** cfs_fail_loc=305, val=0***
[ 4443.346495] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 00:57:52 (1790744272)
[ 4443.929763] Lustre: Mounted lustre-client - version 2.17.59_3_g21e240e
[ 4446.070381] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4456.419824] Lustre: Unmounted lustre-client
[ 4463.577759] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 00:58:12 (1790744292)
[ 4550.189676] LustreError: lustre-MDT0000-mdc-ffff8c3b46b94000: operation mds_reint to node 192.168.206.128@tcp failed: rc = -19
[ 4550.203319] LustreError: Skipped 1 previous similar message
[ 4550.209537] Lustre: lustre-MDT0000-mdc-ffff8c3b46b94000: Connection to lustre-MDT0000 (at 192.168.206.128@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4550.227587] Lustre: Skipped 23 previous similar messages
[ 4567.008169] Lustre: 2356:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790744381/real 1790744381]  req@ffff8c3b58bd1180 x1877726915684224/t0(0) o400->MGC192.168.206.128@tcp@192.168.206.128@tcp:26/25 lens 224/224 e 0 to 1 dl 1790744397 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4567.040137] Lustre: 2356:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 97 previous similar messages
[ 4567.050441] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 192.168.206.128@tcp) was lost; in progress operations using this service will fail
[ 4577.255641] Lustre: Evicted from MGS (at 192.168.206.128@tcp) after server handle changed from 0xcc7f1be12873f344 to 0xcc7f1be12876a955
[ 4577.269533] Lustre: MGC192.168.206.128@tcp: Connection restored to 192.168.206.128@tcp (at 192.168.206.128@tcp)
[ 4577.279802] Lustre: Skipped 24 previous similar messages
[ 4739.430480] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 01:02:47 (1790744567)
[ 4745.206652] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4760.560096] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 192.168.206.128@tcp) was lost; in progress operations using this service will fail
[ 4760.590182] Lustre: Evicted from MGS (at 192.168.206.128@tcp) after server handle changed from 0xcc7f1be12876a955 to 0xcc7f1be1287b72be
[ 4760.682689] LustreError: lustre-MDT0000-mdc-ffff8c3b46b94000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4775.053586] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 01:03:23 (1790744603)
[ 4775.563737] Lustre: Mounted lustre-client - version 2.17.59_3_g21e240e
[ 4783.160477] LustreError: lustre-OST0000-osc-ffff8c3b46b94000: operation ldlm_enqueue to node 192.168.206.128@tcp failed: rc = -107
[ 4783.180433] LustreError: lustre-OST0000-osc-ffff8c3b46b94000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4795.047542] Lustre: Unmounted lustre-client
[ 4803.432384] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 01:03:51 (1790744631)
[ 4810.179693] LustreError: lustre-MDT0000-mdc-ffff8c3b46b94000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4810.219625] LustreError: 61172:0:(file.c:6243:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4820.518562] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 01:04:08 (1790744648)
[ 4820.830504] LustreError: 61873:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4826.080151] LustreError: 61873:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=0
[ 4826.090040] LustreError: 61873:0:(import.c:756:ptlrpc_connect_import_locked()) already connected
[ 4826.361126] LustreError: lustre-MDT0000-mdc-ffff8c3b46b94000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4826.397541] LustreError: 61893:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4826.517421] LustreError: 61898:0:(file.c:6243:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4827.641880] LustreError: 61904:0:(file.c:6243:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4827.662575] LustreError: 61904:0:(file.c:6243:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4829.951784] LustreError: 61927:0:(file.c:6243:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4829.968585] LustreError: 61927:0:(file.c:6243:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4834.795589] LustreError: 61972:0:(file.c:6243:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4834.812939] LustreError: 61972:0:(file.c:6243:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4844.977787] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 01:04:33 (1790744673)
[ 4886.202853] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4887.738322] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4899.026296] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 01:05:27 (1790744727)
[ 4939.222541] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4941.126948] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4951.678643] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 01:06:20 (1790744780)
[ 5012.607922] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5013.897058] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5021.280499] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 01:07:29 (1790744849)
[ 5060.335494] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5061.797601] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5067.467879] Lustre: Unmounted lustre-client
[ 5115.709989] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5118.306320] Lustre: Mounted lustre-client - version 2.17.59_3_g21e240e
[ 5124.540535] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 01:09:12 (1790744952)
[ 5127.660508] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5129.323719] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 01:09:18 (1790744958)
[ 5158.429559] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 01:09:46 (1790744986)
[ 5160.339553] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5162.221446] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 01:09:50 (1790744990)
[ 5163.459906] Lustre: *** cfs_fail_loc=805, val=0***
[ 5163.501948] Lustre: Mounted lustre-client - version 2.17.59_3_g21e240e
[ 5168.172357] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5189.088234] Lustre: 2356:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790745003/real 1790745003]  req@ffff8c3b46587480 x1877726917737600/t0(0) o400->lustre-MDT0000-mdc-ffff8c3b43119800@192.168.206.128@tcp:12/10 lens 224/224 e 0 to 1 dl 1790745019 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5189.093600] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 192.168.206.128@tcp) was lost; in progress operations using this service will fail
[ 5189.116248] Lustre: 2356:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5189.116320] Lustre: lustre-MDT0000-mdc-ffff8c3b43119800: Connection to lustre-MDT0000 (at 192.168.206.128@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5189.131255] Lustre: Evicted from MGS (at 192.168.206.128@tcp) after server handle changed from 0xcc7f1be1287b7cb2 to 0xcc7f1be1287b7ecd
[ 5189.133212] Lustre: Skipped 12 previous similar messages
[ 5189.155738] Lustre: MGC192.168.206.128@tcp: Connection restored to 192.168.206.128@tcp (at 192.168.206.128@tcp)
[ 5189.160435] Lustre: Skipped 12 previous similar messages
[ 5190.889785] Lustre: 68538:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.128@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 5205.788307] Lustre: Unmounted lustre-client
[ 5213.677287] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 01:10:42 (1790745042)
[ 5236.921316] Lustre: 68538:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.128@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 5255.695871] LustreError: 2355:0:(client.c:3449:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff8c3b50f33100 x1877726917749376/t94489280516(94489280516) o101->lustre-MDT0000-mdc-ffff8c3b43119800@192.168.206.128@tcp:12/10 lens 664/608 e 0 to 0 dl 1790745101 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5260.367507] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5261.830498] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5270.032515] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 01:11:38 (1790745098)
[ 5271.138334] LustreError: lustre-OST0000-osc-ffff8c3b43119800: operation ost_write to node 192.168.206.128@tcp failed: rc = -107
[ 5271.149496] LustreError: Skipped 2 previous similar messages
[ 5271.178608] LustreError: lustre-OST0000-osc-ffff8c3b43119800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5271.190964] Lustre: 2356:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.206.128@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5271.208743] LustreError: 73078:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0000-osc-ffff8c3b43119800: namespace resource [0x240000400:0x3d42:0x0].0x0 (ffff8c3b5205e500) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5277.748389] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 01:11:46 (1790745106)
[ 5279.227871] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5280.646768] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 01:11:49 (1790745109)
[ 5281.931269] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5283.534943] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 01:11:52 (1790745112)
[ 5284.961600] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5286.461414] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 01:11:55 (1790745115)
[ 5287.829657] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5289.376815] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 01:11:58 (1790745118)
[ 5290.788332] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5292.186963] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 01:12:01 (1790745121)
[ 5293.432205] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5295.038419] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 01:12:03 (1790745123)
[ 5296.261126] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5297.977986] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 01:12:06 (1790745126)
[ 5299.764666] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5301.276575] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 01:12:10 (1790745130)
[ 5302.650820] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5304.444794] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 01:12:13 (1790745133)
[ 5306.055020] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5307.739899] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 01:12:16 (1790745136)
[ 5309.004986] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5310.921720] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 01:12:19 (1790745139)
[ 5312.932826] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5314.874912] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 01:12:23 (1790745143)
[ 5348.228487] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 01:12:56 (1790745176)
[ 5379.524089] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 01:13:28 (1790745208)
[ 5379.918263] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5391.569866] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 01:13:39 (1790745219)
[ 5391.869552] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5401.573862] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 01:13:50 (1790745230)
[ 5402.259345] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5409.645233] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 01:13:58 (1790745238)
[ 5410.037126] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5418.657475] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 01:14:07 (1790745247)
[ 5419.039939] Lustre: *** cfs_fail_loc=510, val=3***
[ 5427.758198] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 01:14:16 (1790745256)
[ 5428.092247] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5437.953377] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 01:14:26 (1790745266)
[ 5438.279535] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5504.785913] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 01:15:33 (1790745333)
[ 5505.139862] LustreError: 83282:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5507.850332] LustreError: lustre-MDT0000-mdc-ffff8c3b43119800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5507.867165] LustreError: 83296:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff8c3b43119800: inode [0x20000a042:0xf:0x0] mdc close failed: rc = -108
[ 5507.908876] LustreError: 83296:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8c3b43119800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8c3b521a8500) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5509.216145] LustreError: 83282:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5511.318543] LustreError: 83312:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5514.133267] LustreError: lustre-MDT0000-mdc-ffff8c3b43119800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5514.153411] LustreError: 83326:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8c3b43119800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff8c3b60118b00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5514.170400] LustreError: 83326:0:(ldlm_resource.c:1227:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5515.392158] LustreError: 83312:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5522.154853] LustreError: lustre-MDT0000-mdc-ffff8c3b43119800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5522.543177] LustreError: 83368:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5525.323176] LustreError: lustre-MDT0000-mdc-ffff8c3b43119800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5525.407419] LustreError: 83387:0:(file.c:6243:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5525.420418] LustreError: 83387:0:(file.c:6243:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5526.503817] LustreError: 83394:0:(file.c:6243:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5526.524616] LustreError: 83394:0:(file.c:6243:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5526.608829] LustreError: 83368:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5526.616244] LustreError: 83368:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff8c3b43119800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5526.628422] LustreError: 83368:0:(file.c:252:ll_close_inode_openhandle()) Skipped 1 previous similar message
[ 5528.831885] LustreError: 83416:0:(file.c:6243:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5528.850571] LustreError: 83416:0:(file.c:6243:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5532.401888] LustreError: 83443:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5535.232545] LustreError: lustre-MDT0000-mdc-ffff8c3b43119800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5535.251509] LustreError: 83457:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8c3b43119800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8c3b5205eb00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5535.277338] LustreError: 83457:0:(ldlm_resource.c:1227:ldlm_resource_complain()) Skipped 2 previous similar messages
[ 5536.489987] LustreError: 83443:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5543.497285] LustreError: lustre-MDT0000-mdc-ffff8c3b43119800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5543.979613] LustreError: 83499:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5546.983103] LustreError: lustre-MDT0000-mdc-ffff8c3b43119800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5547.012235] LustreError: 83514:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8c3b43119800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8c3b5205e500) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5547.037530] LustreError: 83514:0:(ldlm_resource.c:1227:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5548.048674] LustreError: 83499:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5551.156718] LustreError: lustre-MDT0000-mdc-ffff8c3b43119800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5551.285420] LustreError: 83546:0:(file.c:6243:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5551.293567] LustreError: 83546:0:(file.c:6243:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5552.260361] LustreError: 83527:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff8c3b43119800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5560.573260] LustreError: 83598:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5560.588399] LustreError: 83598:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5564.476229] LustreError: lustre-MDT0000-mdc-ffff8c3b43119800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5564.573140] LustreError: 83621:0:(file.c:6243:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5564.582512] LustreError: 83621:0:(file.c:6243:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5564.583059] LustreError: 83598:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5564.606924] LustreError: 83598:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5564.620503] LustreError: 83598:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff8c3b43119800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5577.274751] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 01:16:45 (1790745405)
[ 5596.504236] LustreError: 70957:0:(ldlm_lockd.c:2893:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5596.522365] LustreError: 70957:0:(ldlm_lockd.c:2893:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5600.608171] LustreError: 70957:0:(ldlm_lockd.c:2893:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5600.620804] LustreError: 70957:0:(ldlm_lockd.c:2893:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5608.546182] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 01:17:17 (1790745437)
[ 5658.646971] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 01:18:07 (1790745487)
[ 5723.310638] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 01:19:11 (1790745551)
[ 5756.113517] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 01:19:44 (1790745584)
[ 5756.651438] Lustre: Mounted lustre-client - version 2.17.59_3_g21e240e
[ 5879.715672] Lustre: Unmounted lustre-client
[ 5886.153886] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 01:21:54 (1790745714)
[ 5886.664417] LustreError: 87926:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5888.996520] LustreError: lustre-OST0000-osc-ffff8c3b43119800: operation ldlm_enqueue to node 192.168.206.128@tcp failed: rc = -107
[ 5889.001756] LustreError: Skipped 8 previous similar messages
[ 5889.007638] Lustre: lustre-OST0000-osc-ffff8c3b43119800: Connection to lustre-OST0000 (at 192.168.206.128@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5889.018541] Lustre: Skipped 17 previous similar messages
[ 5889.045176] LustreError: lustre-OST0000-osc-ffff8c3b43119800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5889.065096] LustreError: 88008:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0000-osc-ffff8c3b43119800: namespace resource [0x240000400:0x3d6d:0x0].0x0 (ffff8c3b60118500) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5889.097961] LustreError: 88008:0:(ldlm_resource.c:1227:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5889.392129] LustreError: 87926:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5889.400524] Lustre: 2356:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.206.128@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5895.247877] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 01:22:03 (1790745723)
[ 5952.481093] Lustre: 88592:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790745727/real 1790745727]  req@ffff8c3b58a8b480 x1877726917922560/t0(0) o101->lustre-MDT0000-mdc-ffff8c3b43119800@192.168.206.128@tcp:12/10 lens 328/344 e 0 to 1 dl 1790745782 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 5952.528561] Lustre: 88592:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 20 previous similar messages
[ 5952.557673] Lustre: lustre-MDT0000-mdc-ffff8c3b43119800: Connection restored to 192.168.206.128@tcp (at 192.168.206.128@tcp)
[ 5952.566466] Lustre: Skipped 18 previous similar messages
[ 5959.975280] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 01:23:08 (1790745788)
[ 5961.342706] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 5962.942943] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 01:23:11 (1790745791)
[ 6027.555700] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6029.386979] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 01:24:17 (1790745857)
[ 6087.285777] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 01:25:15 (1790745915)
[ 6089.457606] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6091.336668] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 01:25:19 (1790745919)
[ 6093.056494] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6094.849591] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 01:25:23 (1790745923)
[ 6126.095609] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 01:25:54 (1790745954)
[ 6136.887435] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6162.426515] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 192.168.206.128@tcp) was lost; in progress operations using this service will fail
[ 6162.438398] LustreError: Skipped 2 previous similar messages
[ 6162.458780] Lustre: Evicted from MGS (at 192.168.206.128@tcp) after server handle changed from 0xcc7f1be1287b8b29 to 0xcc7f1be1287ba44a
[ 6162.470333] Lustre: Skipped 2 previous similar messages
[ 6175.202351] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6177.018180] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6186.138900] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 01:26:54 (1790746014)
[ 6188.313530] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6190.124906] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 01:26:58 (1790746018)
[ 6207.489478] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 192.168.206.128@tcp) was lost; in progress operations using this service will fail
[ 6207.543216] Lustre: Evicted from MGS (at 192.168.206.128@tcp) after server handle changed from 0xcc7f1be1287ba44a to 0xcc7f1be1287ba802
[ 6219.822751] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 01:27:28 (1790746048)
[ 6268.457217] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 01:28:16 (1790746096)
[ 6323.492805] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6326.842547] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6411.236898] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 192.168.206.128@tcp) was lost; in progress operations using this service will fail
[ 6411.261972] LustreError: Skipped 1 previous similar message
[ 6421.502843] Lustre: Evicted from MGS (at 192.168.206.128@tcp) after server handle changed from 0xcc7f1be1287baa0f to 0xcc7f1be1287bb85c
[ 6421.527820] Lustre: Skipped 1 previous similar message
[ 6423.022860] Lustre: 68538:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.128@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6438.320246] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6440.038749] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6476.363281] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6478.026866] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6616.745543] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 01:34:05 (1790746445)
[ 6624.517026] LustreError: lustre-OST0000-osc-ffff8c3b43119800: operation ost_setattr to node 192.168.206.128@tcp failed: rc = -19
[ 6624.524425] LustreError: Skipped 51 previous similar messages
[ 6624.531102] Lustre: lustre-OST0000-osc-ffff8c3b43119800: Connection to lustre-OST0000 (at 192.168.206.128@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6624.544819] Lustre: Skipped 9 previous similar messages
[ 6677.031342] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6681.696482] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6924.745103] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 01:39:13 (1790746753)
[ 7072.237381] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 192.168.206.128@tcp) was lost; in progress operations using this service will fail
[ 7072.251285] LustreError: Skipped 1 previous similar message
[ 7072.273994] Lustre: Evicted from MGS (at 192.168.206.128@tcp) after server handle changed from 0xcc7f1be1287bbd6b to 0xcc7f1be1288b444c
[ 7072.294967] Lustre: Skipped 1 previous similar message
[ 7072.303425] Lustre: MGC192.168.206.128@tcp: Connection restored to 192.168.206.128@tcp (at 192.168.206.128@tcp)
[ 7072.316310] Lustre: Skipped 12 previous similar messages
[ 7109.571796] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 01:42:18 (1790746938)
[ 7111.226464] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7113.002661] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 01:42:21 (1790746941)
[ 7115.079429] LustreError: lustre-MDT0000-mdc-ffff8c3b43119800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7122.462871] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 01:42:31 (1790746951)
[ 7289.327305] LustreError: lustre-OST0000-osc-ffff8c3b43119800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7291.569879] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 01:45:19 (1790747119)
[ 7316.834403] Lustre: 2359:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790747126/real 1790747126]  req@ffff8c3b46d0c380 x1877726935502208/t0(0) o4->lustre-OST0000-osc-ffff8c3b43119800@192.168.206.128@tcp:6/4 lens 4584/448 e 0 to 1 dl 1790747146 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7316.864913] Lustre: 2359:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 24 previous similar messages
[ 7316.870229] Lustre: lustre-OST0000-osc-ffff8c3b43119800: Connection to lustre-OST0000 (at 192.168.206.128@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7316.880913] Lustre: Skipped 3 previous similar messages
[ 7335.782574] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 01:46:04 (1790747164)
[ 7337.116879] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7338.659769] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 01:46:07 (1790747167)
[ 7340.101568] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7341.859829] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 01:46:10 (1790747170)
[ 7378.102867] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 01:46:46 (1790747206)
[ 7419.378286] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 192.168.206.128@tcp) was lost; in progress operations using this service will fail
[ 7419.401457] Lustre: Evicted from MGS (at 192.168.206.128@tcp) after server handle changed from 0xcc7f1be1288b444c to 0xcc7f1be1288b6b97
[ 7419.415350] Lustre: Skipped 1 previous similar message
[ 7432.531481] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 01:47:40 (1790747260)
[ 7434.537747] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7436.332797] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 01:47:44 (1790747264)
[ 7437.598796] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7439.175954] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 01:47:47 (1790747267)
[ 7448.092854] Lustre: Unmounted lustre-client
[ 7449.383658] Lustre: Mounted lustre-client - version 2.17.59_3_g21e240e
[ 7451.966852] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7470.049140] Lustre: 2357:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790747284/real 1790747284]  req@ffff8c3b46cdc000 x1877726935688832/t0(0) o400->MGC192.168.206.128@tcp@192.168.206.128@tcp:26/25 lens 224/224 e 0 to 1 dl 1790747300 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7480.031488] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2