[ 1731.447385] LustreError: lustre-OST0000-osc-ffff8e71c50fa000: operation ost_sync to node 192.168.206.113@tcp failed: rc = -107
[ 1731.469539] LustreError: lustre-OST0000-osc-ffff8e71c50fa000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1731.490476] Lustre: 2368:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.206.113@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1741.271299] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 10:59:41 (1788793181)
[ 1745.073496] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1747.009911] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 10:59:47 (1788793187)
[ 1749.497372] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1752.074085] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 10:59:52 (1788793192)
[ 1755.915714] LustreError: lustre-MDT0000-mdc-ffff8e71c50fa000: operation ldlm_enqueue to node 192.168.206.113@tcp failed: rc = -19
[ 1776.123127] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 192.168.206.113@tcp) was lost; in progress operations using this service will fail
[ 1776.151930] Lustre: Evicted from MGS (at 192.168.206.113@tcp) after server handle changed from 0x540e7fb81284de28 to 0x540e7fb81284f662
[ 1883.161092] LustreError: lustre-MDT0000-mdc-ffff8e71c50fa000: operation ldlm_enqueue to node 192.168.206.113@tcp failed: rc = -19
[ 1900.000552] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 192.168.206.113@tcp) was lost; in progress operations using this service will fail
[ 1910.329416] Lustre: Evicted from MGS (at 192.168.206.113@tcp) after server handle changed from 0x540e7fb81284f662 to 0x540e7fb81287040a
[ 1918.887398] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 11:02:39 (1788793359)
[ 1919.371081] Lustre: *** cfs_fail_loc=305, val=0***
[ 1919.373963] Lustre: Skipped 4 previous similar messages
[ 1940.124445] LustreError: lustre-OST0000-osc-ffff8e71c50fa000: operation ost_connect to node 192.168.206.113@tcp failed: rc = -75
[ 1967.073579] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 192.168.206.113@tcp) was lost; in progress operations using this service will fail
[ 1977.326300] Lustre: lustre-MDT0000-mdc-ffff8e71c50fa000: Connection to lustre-MDT0000 (at 192.168.206.113@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 1977.331623] Lustre: Evicted from MGS (at 192.168.206.113@tcp) after server handle changed from 0x540e7fb81287040a to 0x540e7fb812870fc5
[ 1977.366085] Lustre: Skipped 11 previous similar messages
[ 1977.398589] Lustre: MGC192.168.206.113@tcp: Connection restored to 192.168.206.113@tcp (at 192.168.206.113@tcp)
[ 1977.410751] Lustre: Skipped 13 previous similar messages
[ 1993.162421] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 1994.797749] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2003.807954] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 11:04:04 (1788793444)
[ 2018.292875] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 192.168.206.113@tcp) was lost; in progress operations using this service will fail
[ 2018.319089] Lustre: Evicted from MGS (at 192.168.206.113@tcp) after server handle changed from 0x540e7fb812870fc5 to 0x540e7fb812871544
[ 2018.434528] LustreError: lustre-MDT0000-mdc-ffff8e71c50fa000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2041.532651] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 11:04:42 (1788793482)
[ 2073.868286] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 11:05:14 (1788793514)
[ 2089.036957] LustreError: lustre-MDT0000-mdc-ffff8e71c50fa000: operation mds_reint to node 192.168.206.113@tcp failed: rc = -19
[ 2089.056699] LustreError: Skipped 1 previous similar message
[ 2106.335783] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 192.168.206.113@tcp) was lost; in progress operations using this service will fail
[ 2116.681933] Lustre: Evicted from MGS (at 192.168.206.113@tcp) after server handle changed from 0x540e7fb812871544 to 0x540e7fb8128765ee
[ 2118.600384] Lustre: 15908:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.113@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2142.944942] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2145.990823] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2212.657186] LustreError: lustre-MDT0000-mdc-ffff8e71c50fa000: operation mds_reint to node 192.168.206.113@tcp failed: rc = -19
[ 2232.799210] Lustre: 2371:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788793659/real 1788793659]  req@ffff8e71d01a7480 x1875683827994240/t0(0) o400->MGC192.168.206.113@tcp@192.168.206.113@tcp:26/25 lens 224/224 e 0 to 1 dl 1788793675 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2232.835901] Lustre: 2371:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 14 previous similar messages
[ 2232.849681] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 192.168.206.113@tcp) was lost; in progress operations using this service will fail
[ 2232.893254] Lustre: Evicted from MGS (at 192.168.206.113@tcp) after server handle changed from 0x540e7fb8128765ee to 0x540e7fb81288b792
[ 2262.140802] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2264.514052] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2330.386078] LustreError: lustre-MDT0000-mdc-ffff8e71c50fa000: operation ldlm_enqueue to node 192.168.206.113@tcp failed: rc = -19
[ 2350.543582] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 192.168.206.113@tcp) was lost; in progress operations using this service will fail
[ 2350.577214] Lustre: Evicted from MGS (at 192.168.206.113@tcp) after server handle changed from 0x540e7fb81288b792 to 0x540e7fb8128a666d
[ 2378.298586] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2380.254481] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2410.671759] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 11:10:51 (1788793851)
[ 2415.888078] LustreError: lustre-MDT0000-mdc-ffff8e71c50fa000: operation ldlm_enqueue to node 192.168.206.113@tcp failed: rc = -19
[ 2415.902334] LustreError: Skipped 2 previous similar messages
[ 2436.575417] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 192.168.206.113@tcp) was lost; in progress operations using this service will fail
[ 2436.610324] Lustre: Evicted from MGS (at 192.168.206.113@tcp) after server handle changed from 0x540e7fb8128a666d to 0x540e7fb8128b36fa
[ 2437.832917] Lustre: 15908:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.113@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2453.529794] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2478.256890] Lustre: 15908:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.113@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2494.935317] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2540.833646] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2554.156115] LustreError: lustre-MDT0000-mdc-ffff8e71c50fa000: operation ldlm_enqueue to node 192.168.206.113@tcp failed: rc = -19
[ 2554.168996] LustreError: Skipped 3 previous similar messages
[ 2572.256116] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 192.168.206.113@tcp) was lost; in progress operations using this service will fail
[ 2572.271891] LustreError: Skipped 2 previous similar messages
[ 2582.525593] Lustre: Evicted from MGS (at 192.168.206.113@tcp) after server handle changed from 0x540e7fb8128b71d9 to 0x540e7fb8128bbd12
[ 2582.537924] Lustre: Skipped 2 previous similar messages
[ 2582.545455] Lustre: MGC192.168.206.113@tcp: Connection restored to 192.168.206.113@tcp (at 192.168.206.113@tcp)
[ 2582.552665] Lustre: Skipped 15 previous similar messages
[ 2590.321400] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2614.765892] Lustre: lustre-MDT0000-mdc-ffff8e71c50fa000: Connection to lustre-MDT0000 (at 192.168.206.113@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2614.781872] Lustre: Skipped 8 previous similar messages
[ 2650.614213] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2716.098152] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2818.442525] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 11:17:39 (1788794259)
[ 2834.211259] LustreError: lustre-OST0000-osc-ffff8e71c50fa000: operation ldlm_enqueue to node 192.168.206.113@tcp failed: rc = -19
[ 2834.261594] LustreError: Skipped 3 previous similar messages
[ 2882.800460] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 2885.857398] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3182.723545] Lustre: lustre-OST0000-osc-ffff8e71c50fa000: Connection restored to 192.168.206.113@tcp (at 192.168.206.113@tcp)
[ 3182.731728] Lustre: Skipped 7 previous similar messages
[ 3196.171525] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3197.816515] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3494.030786] LustreError: lustre-OST0000-osc-ffff8e71c50fa000: operation ldlm_enqueue to node 192.168.206.113@tcp failed: rc = -19
[ 3494.047695] LustreError: Skipped 1 previous similar message
[ 3494.053942] Lustre: lustre-OST0000-osc-ffff8e71c50fa000: Connection to lustre-OST0000 (at 192.168.206.113@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3494.065385] Lustre: Skipped 4 previous similar messages
[ 3532.446790] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3534.641950] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3789.528669] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 11:33:50 (1788795230)
[ 3807.200369] Lustre: 50561:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788795233/real 1788795233]  req@ffff8e71d1f60380 x1875683852347008/t0(0) o101->lustre-MDT0000-mdc-ffff8e71c50fa000@192.168.206.113@tcp:12/10 lens 576/1152 e 0 to 1 dl 1788795249 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3807.224924] Lustre: 50561:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
[ 3807.246735] Lustre: lustre-MDT0000-mdc-ffff8e71c50fa000: Connection restored to 192.168.206.113@tcp (at 192.168.206.113@tcp)
[ 3807.252678] Lustre: Skipped 1 previous similar message
[ 3814.490385] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 11:34:15 (1788795255)
[ 3838.175603] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 11:34:39 (1788795279)
[ 3861.778333] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 11:35:02 (1788795302)
[ 3862.302149] Lustre: Mounted lustre-client - version 2.17.57_82_gfdb2c0c
[ 3893.232481] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 192.168.206.113@tcp) was lost; in progress operations using this service will fail
[ 3893.246436] LustreError: Skipped 3 previous similar messages
[ 3893.269630] Lustre: Evicted from MGS (at 192.168.206.113@tcp) after server handle changed from 0x540e7fb8128d7229 to 0x540e7fb812a54a29
[ 3893.277958] Lustre: Skipped 3 previous similar messages
[ 3894.239292] Lustre: 2369:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788795320/real 1788795320]  req@ffff8e71d03e5880 x1875683852366208/t0(0) o400->lustre-MDT0000-mdc-ffff8e71c8063800@192.168.206.113@tcp:12/10 lens 224/224 e 0 to 1 dl 1788795336 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3894.268569] Lustre: 2369:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 3903.285702] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 3904.584819] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3906.645990] Lustre: Unmounted lustre-client
[ 3911.638647] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 11:35:52 (1788795352)
[ 4048.863259] Lustre: 2371:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788795475/real 1788795475]  req@ffff8e71d0207100 x1875683852398464/t0(0) o4->lustre-OST0000-osc-ffff8e71c50fa000@192.168.206.113@tcp:6/4 lens 488/448 e 0 to 1 dl 1788795491 ref 2 fl Rpc:XQr/602/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4048.907847] Lustre: 2371:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 73 previous similar messages
[ 4096.991376] Lustre: lustre-OST0000-osc-ffff8e71c50fa000: Connection to lustre-OST0000 (at 192.168.206.113@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4097.011140] Lustre: Skipped 15 previous similar messages
[ 4206.415268] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 11:40:47 (1788795647)
[ 4255.342791] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 11:41:36 (1788795696)
[ 4257.823095] Lustre: Unmounted lustre-client
[ 4276.975445] Lustre: Mounted lustre-client - version 2.17.57_82_gfdb2c0c
[ 4284.631543] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 11:42:05 (1788795725)
[ 4284.863170] LustreError: 56484:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4285.924301] LustreError: 56484:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4286.143622] Lustre: *** cfs_fail_loc=305, val=0***
[ 4308.671220] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 11:42:29 (1788795749)
[ 4309.109101] Lustre: Mounted lustre-client - version 2.17.57_82_gfdb2c0c
[ 4311.176544] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4321.499993] Lustre: Unmounted lustre-client
[ 4326.944381] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 11:42:48 (1788795768)
[ 4413.300461] LustreError: lustre-MDT0000-mdc-ffff8e71d016b800: operation mds_reint to node 192.168.206.113@tcp failed: rc = -19
[ 4432.355978] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 192.168.206.113@tcp) was lost; in progress operations using this service will fail
[ 4432.383910] Lustre: Evicted from MGS (at 192.168.206.113@tcp) after server handle changed from 0x540e7fb812a562be to 0x540e7fb812a817c5
[ 4432.399770] Lustre: MGC192.168.206.113@tcp: Connection restored to 192.168.206.113@tcp (at 192.168.206.113@tcp)
[ 4432.408621] Lustre: Skipped 24 previous similar messages
[ 4434.033614] Lustre: 55882:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.113@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 4594.184895] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 11:47:15 (1788796035)
[ 4599.102467] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4615.662224] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 192.168.206.113@tcp) was lost; in progress operations using this service will fail
[ 4615.688746] Lustre: Evicted from MGS (at 192.168.206.113@tcp) after server handle changed from 0x540e7fb812a817c5 to 0x540e7fb812ace063
[ 4615.713348] LustreError: lustre-MDT0000-mdc-ffff8e71d016b800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4630.728440] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 11:47:51 (1788796071)
[ 4631.238845] Lustre: Mounted lustre-client - version 2.17.57_82_gfdb2c0c
[ 4638.519908] LustreError: lustre-OST0000-osc-ffff8e71d016b800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4638.559891] LustreError: 60201:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff8e71d016b800: namespace resource [0x240000400:0x4303:0x0].0x0 (ffff8e71c6ee3200) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4647.903143] Lustre: 2371:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788796074/real 1788796074]  req@ffff8e71c4586300 x1875683857053312/t0(0) o103->lustre-OST0000-osc-ffff8e71d2039800@192.168.206.113@tcp:17/18 lens 328/224 e 0 to 1 dl 1788796090 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295
[ 4647.928733] Lustre: 2371:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 69 previous similar messages
[ 4651.055409] Lustre: Unmounted lustre-client
[ 4657.450212] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 11:48:18 (1788796098)
[ 4664.197228] LustreError: lustre-MDT0000-mdc-ffff8e71d016b800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4664.216990] LustreError: 60873:0:(file.c:6216:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4671.446333] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 11:48:32 (1788796112)
[ 4671.833781] LustreError: 61576:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4675.845744] LustreError: lustre-MDT0000-mdc-ffff8e71d016b800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4675.857919] LustreError: 61593:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4675.864056] LustreError: 61576:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=981
[ 4675.869258] LustreError: 61576:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting
[ 4675.918220] LustreError: 61598:0:(file.c:6216:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4676.996393] LustreError: 61604:0:(file.c:6216:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4677.008198] LustreError: 61604:0:(file.c:6216:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4679.281380] LustreError: 61628:0:(file.c:6216:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4679.289018] LustreError: 61628:0:(file.c:6216:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4683.830512] LustreError: 61673:0:(file.c:6216:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4683.853067] LustreError: 61673:0:(file.c:6216:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4694.131926] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 11:48:55 (1788796135)
[ 4701.164718] Lustre: lustre-OST0000-osc-ffff8e71d016b800: Connection to lustre-OST0000 (at 192.168.206.113@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4701.192954] Lustre: Skipped 17 previous similar messages
[ 4728.715846] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4729.738384] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4738.252695] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 11:49:39 (1788796179)
[ 4771.535776] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4772.974541] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4782.341570] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 11:50:23 (1788796223)
[ 4846.829473] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4848.395720] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4856.308337] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 11:51:37 (1788796297)
[ 4891.455459] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4893.334528] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4899.930750] Lustre: Unmounted lustre-client
[ 4958.600814] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4961.170532] Lustre: Mounted lustre-client - version 2.17.57_82_gfdb2c0c
[ 4967.368759] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 11:53:28 (1788796408)
[ 4969.796073] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 4971.129496] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 11:53:32 (1788796412)
[ 4999.929818] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 11:54:00 (1788796440)
[ 5001.635482] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5003.305527] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 11:54:04 (1788796444)
[ 5004.633266] Lustre: *** cfs_fail_loc=805, val=0***
[ 5004.681694] Lustre: Mounted lustre-client - version 2.17.57_82_gfdb2c0c
[ 5009.862980] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5031.392943] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 192.168.206.113@tcp) was lost; in progress operations using this service will fail
[ 5041.642548] Lustre: Evicted from MGS (at 192.168.206.113@tcp) after server handle changed from 0x540e7fb812acea96 to 0x540e7fb812acee40
[ 5041.666095] Lustre: MGC192.168.206.113@tcp: Connection restored to 192.168.206.113@tcp (at 192.168.206.113@tcp)
[ 5041.671421] LustreError: 2367:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8e71c7d99800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff8e71c7ebe900) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5041.673056] Lustre: Skipped 13 previous similar messages
[ 5044.213072] Lustre: Unmounted lustre-client
[ 5055.175264] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 11:54:55 (1788796495)
[ 5085.422728] Lustre: 68250:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.113@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 5099.039167] LustreError: 2367:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff8e71c44b7b80 x1875683857133952/t94489280517(94489280517) o101->lustre-MDT0000-mdc-ffff8e71d2aa4000@192.168.206.113@tcp:12/10 lens 648/608 e 0 to 0 dl 1788796557 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5104.382198] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5105.885624] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5112.895602] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 11:55:54 (1788796554)
[ 5114.348081] LustreError: lustre-OST0000-osc-ffff8e71d2aa4000: operation ost_write to node 192.168.206.113@tcp failed: rc = -107
[ 5114.351840] LustreError: Skipped 3 previous similar messages
[ 5114.362259] LustreError: lustre-OST0000-osc-ffff8e71d2aa4000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5114.377815] Lustre: 2370:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.206.113@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5114.418751] LustreError: 72787:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff8e71d2aa4000: namespace resource [0x240000400:0x4362:0x0].0x0 (ffff8e71c6ee3500) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5114.439740] LustreError: 72787:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5123.050515] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 11:56:03 (1788796563)
[ 5124.785614] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5126.890822] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 11:56:07 (1788796567)
[ 5128.325380] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5130.039810] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 11:56:10 (1788796570)
[ 5131.654885] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5133.153379] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 11:56:14 (1788796574)
[ 5134.709645] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5136.337386] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 11:56:17 (1788796577)
[ 5137.690789] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5139.455176] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 11:56:20 (1788796580)
[ 5141.028771] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5142.403339] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 11:56:23 (1788796583)
[ 5143.365157] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5144.778854] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 11:56:25 (1788796585)
[ 5146.195031] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5147.826256] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 11:56:29 (1788796589)
[ 5149.035217] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5150.575392] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 11:56:31 (1788796591)
[ 5152.048525] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5153.873162] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 11:56:34 (1788796594)
[ 5156.378602] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5158.227696] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 11:56:39 (1788796599)
[ 5160.914152] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5162.580702] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 11:56:43 (1788796603)
[ 5193.707772] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 11:57:14 (1788796634)
[ 5221.597312] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 11:57:42 (1788796662)
[ 5221.906677] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5230.732352] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 11:57:51 (1788796671)
[ 5231.242329] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5240.109983] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 11:58:01 (1788796681)
[ 5240.423751] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5247.244907] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 11:58:08 (1788796688)
[ 5247.425850] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5253.713373] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 11:58:15 (1788796695)
[ 5254.034116] Lustre: *** cfs_fail_loc=510, val=3***
[ 5261.284466] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 11:58:22 (1788796702)
[ 5261.678570] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5321.055397] Lustre: 2369:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788796708/real 1788796708]  req@ffff8e71c44b6d80 x1875683857174144/t0(0) o3->lustre-OST0000-osc-ffff8e71d2aa4000@192.168.206.113@tcp:6/4 lens 488/4536 e 0 to 1 dl 1788796763 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 5321.089233] Lustre: 2369:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 13 previous similar messages
[ 5321.098188] Lustre: lustre-OST0000-osc-ffff8e71d2aa4000: Connection to lustre-OST0000 (at 192.168.206.113@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5321.118560] Lustre: Skipped 9 previous similar messages
[ 5327.406741] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 11:59:28 (1788796768)
[ 5327.793074] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5395.471720] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 12:00:35 (1788796835)
[ 5395.942637] LustreError: 82998:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5398.647133] LustreError: lustre-MDT0000-mdc-ffff8e71d2aa4000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5398.662541] LustreError: 83013:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8e71d2aa4000: inode [0x20000a042:0xe:0x0] mdc close failed: rc = -108
[ 5398.702419] LustreError: 83013:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8e71d2aa4000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8e71c6ef7200) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5400.039126] LustreError: 82998:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5402.179410] LustreError: 83029:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5405.013830] LustreError: lustre-MDT0000-mdc-ffff8e71d2aa4000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5405.035830] LustreError: 83043:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8e71d2aa4000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8e71c6ee3600) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5405.063788] LustreError: 83043:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5406.271478] LustreError: 83029:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5413.240126] LustreError: lustre-MDT0000-mdc-ffff8e71d2aa4000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5413.800713] LustreError: 83084:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5416.643702] LustreError: lustre-MDT0000-mdc-ffff8e71d2aa4000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5416.735186] LustreError: 83104:0:(file.c:6216:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5416.751696] LustreError: 83104:0:(file.c:6216:ll_inode_revalidate_fini()) Skipped 20 previous similar messages
[ 5417.852064] LustreError: 83110:0:(file.c:6216:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5417.864838] LustreError: 83110:0:(file.c:6216:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5417.880060] LustreError: 83084:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5417.895255] LustreError: 83084:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8e71d2aa4000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5417.917940] LustreError: 83084:0:(file.c:251:ll_close_inode_openhandle()) Skipped 1 previous similar message
[ 5420.273665] LustreError: 83143:0:(file.c:6216:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5420.292260] LustreError: 83143:0:(file.c:6216:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5423.970196] LustreError: 83170:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5426.758333] LustreError: lustre-MDT0000-mdc-ffff8e71d2aa4000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5426.772890] LustreError: 83184:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8e71d2aa4000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8e71c6ef7300) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5426.789610] LustreError: 83184:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5428.063192] LustreError: 83170:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5435.173409] LustreError: lustre-MDT0000-mdc-ffff8e71d2aa4000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5435.552316] LustreError: 83226:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5438.258133] LustreError: lustre-MDT0000-mdc-ffff8e71d2aa4000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5439.631245] LustreError: 83226:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5442.602693] LustreError: lustre-MDT0000-mdc-ffff8e71d2aa4000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5442.678501] LustreError: 83273:0:(file.c:6216:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5442.688468] LustreError: 83273:0:(file.c:6216:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5443.825414] LustreError: 83253:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8e71d2aa4000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5451.640447] LustreError: 83324:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5451.654426] LustreError: 83324:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5454.985574] LustreError: lustre-MDT0000-mdc-ffff8e71d2aa4000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5455.049312] LustreError: 83346:0:(file.c:6216:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5455.056653] LustreError: 83346:0:(file.c:6216:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5455.738205] LustreError: 83324:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5455.750183] LustreError: 83324:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5455.761728] LustreError: 83324:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8e71d2aa4000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5465.957793] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 12:01:46 (1788796906)
[ 5484.391550] LustreError: 79743:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5484.398929] LustreError: 79743:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5488.487283] LustreError: 79743:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5488.489984] LustreError: 79743:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5495.105614] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 12:02:15 (1788796935)
[ 5547.454335] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 12:03:08 (1788796988)
[ 5613.371157] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 12:04:13 (1788797053)
[ 5646.694000] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 12:04:47 (1788797087)
[ 5647.483736] Lustre: Mounted lustre-client - version 2.17.57_82_gfdb2c0c
[ 5771.080498] Lustre: Unmounted lustre-client
[ 5778.359397] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 12:06:59 (1788797219)
[ 5779.037654] LustreError: 87652:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5781.562437] LustreError: lustre-OST0000-osc-ffff8e71d2aa4000: operation ldlm_enqueue to node 192.168.206.113@tcp failed: rc = -107
[ 5781.571236] LustreError: Skipped 9 previous similar messages
[ 5781.586815] LustreError: lustre-OST0000-osc-ffff8e71d2aa4000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5781.605644] LustreError: 87734:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff8e71d2aa4000: namespace resource [0x240000400:0x438d:0x0].0x0 (ffff8e71c6ee3600) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5781.627917] LustreError: 87734:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 3 previous similar messages
[ 5781.887141] LustreError: 87652:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5781.895568] Lustre: 2371:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.206.113@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5788.604816] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 12:07:09 (1788797229)
[ 5845.995695] Lustre: lustre-MDT0000-mdc-ffff8e71d2aa4000: Connection restored to 192.168.206.113@tcp (at 192.168.206.113@tcp)
[ 5846.006888] Lustre: Skipped 18 previous similar messages
[ 5852.026494] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 12:08:13 (1788797293)
[ 5853.494681] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 5855.270686] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 12:08:16 (1788797296)
[ 5918.009912] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 5919.619714] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 12:09:20 (1788797360)
[ 5976.031165] Lustre: 90051:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788797363/real 1788797363]  req@ffff8e71c4584a80 x1875683857330560/t0(0) o36->lustre-MDT0000-mdc-ffff8e71d2aa4000@192.168.206.113@tcp:12/10 lens 496/440 e 0 to 1 dl 1788797418 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ln.0' uid:0 gid:0 projid:4294967295
[ 5976.066992] Lustre: 90051:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
[ 5976.075567] Lustre: lustre-MDT0000-mdc-ffff8e71d2aa4000: Connection to lustre-MDT0000 (at 192.168.206.113@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5976.086956] Lustre: Skipped 16 previous similar messages
[ 5977.425701] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 12:10:18 (1788797418)
[ 5979.444464] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 5980.936594] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 12:10:22 (1788797422)
[ 5982.241449] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 5983.772942] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 12:10:24 (1788797424)
[ 6016.889293] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 12:10:57 (1788797457)
[ 6028.229697] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6052.833537] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 192.168.206.113@tcp) was lost; in progress operations using this service will fail
[ 6052.845093] LustreError: Skipped 2 previous similar messages
[ 6052.869569] Lustre: Evicted from MGS (at 192.168.206.113@tcp) after server handle changed from 0x540e7fb812acf7a8 to 0x540e7fb812ad123c
[ 6052.885672] Lustre: Skipped 2 previous similar messages
[ 6064.391595] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6065.579440] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6074.642368] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 12:11:55 (1788797515)
[ 6076.459442] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6077.663628] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 12:11:59 (1788797519)
[ 6088.685932] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 192.168.206.113@tcp) was lost; in progress operations using this service will fail
[ 6088.717587] Lustre: Evicted from MGS (at 192.168.206.113@tcp) after server handle changed from 0x540e7fb812ad123c to 0x540e7fb812ad1576
[ 6099.621362] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 12:12:20 (1788797540)
[ 6121.333839] Lustre: 68250:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.113@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6143.231744] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 12:13:04 (1788797584)
[ 6194.168597] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6197.186279] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6283.250213] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 192.168.206.113@tcp) was lost; in progress operations using this service will fail
[ 6283.261709] LustreError: Skipped 1 previous similar message
[ 6283.276265] Lustre: Evicted from MGS (at 192.168.206.113@tcp) after server handle changed from 0x540e7fb812ad1801 to 0x540e7fb812ad27eb
[ 6283.292925] Lustre: Skipped 1 previous similar message
[ 6293.553093] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6294.868369] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6325.289562] Lustre: 68250:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.113@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6340.380967] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6342.009240] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6461.709783] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 12:18:22 (1788797902)
[ 6469.001428] LustreError: lustre-OST0000-osc-ffff8e71d2aa4000: operation ost_setattr to node 192.168.206.113@tcp failed: rc = -19
[ 6469.014899] LustreError: Skipped 2 previous similar messages
[ 6516.856313] Lustre: DEBUG MARKER: oleg613-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6519.514975] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6755.967831] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 12:23:16 (1788798196)
[ 6928.362024] Lustre: lustre-MDT0000-mdc-ffff8e71d2aa4000: Connection to lustre-MDT0000 (at 192.168.206.113@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6928.389780] Lustre: Skipped 7 previous similar messages
[ 6938.603264] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 192.168.206.113@tcp) was lost; in progress operations using this service will fail
[ 6938.611631] LustreError: Skipped 1 previous similar message
[ 6938.619244] Lustre: Evicted from MGS (at 192.168.206.113@tcp) after server handle changed from 0x540e7fb812ad299d to 0x540e7fb812bc42e6
[ 6938.629587] Lustre: Skipped 1 previous similar message
[ 6938.634740] Lustre: MGC192.168.206.113@tcp: Connection restored to 192.168.206.113@tcp (at 192.168.206.113@tcp)
[ 6938.641127] Lustre: Skipped 12 previous similar messages
[ 6978.123289] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 12:26:59 (1788798419)
[ 6979.562630] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 6981.407456] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 12:27:02 (1788798422)
[ 6983.259286] LustreError: lustre-MDT0000-mdc-ffff8e71d2aa4000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6991.377756] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 12:27:12 (1788798432)
[ 7158.756698] LustreError: lustre-OST0000-osc-ffff8e71d2aa4000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7160.887097] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 12:30:01 (1788798601)
[ 7186.335325] Lustre: 2368:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788798608/real 1788798608]  req@ffff8e71c8614700 x1875683874365952/t0(0) o4->lustre-OST0000-osc-ffff8e71d2aa4000@192.168.206.113@tcp:6/4 lens 4584/448 e 0 to 1 dl 1788798628 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7186.378508] Lustre: 2368:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 17 previous similar messages
[ 7202.036482] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 12:30:43 (1788798643)
[ 7203.276637] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7205.496411] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 12:30:46 (1788798646)
[ 7206.973284] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7208.882909] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 12:30:49 (1788798649)
[ 7242.586665] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 12:31:23 (1788798683)
[ 7279.590242] LustreError: MGC192.168.206.113@tcp: Connection to MGS (at 192.168.206.113@tcp) was lost; in progress operations using this service will fail
[ 7279.632398] Lustre: Evicted from MGS (at 192.168.206.113@tcp) after server handle changed from 0x540e7fb812bc42e6 to 0x540e7fb812bc6d72
[ 7279.639856] Lustre: Skipped 1 previous similar message
[ 7295.869764] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 12:32:16 (1788798736)
[ 7297.219348] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7298.763228] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 12:32:19 (1788798739)
[ 7300.196961] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7301.912657] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 12:32:22 (1788798742)
[ 7312.719837] Lustre: Unmounted lustre-client
[ 7314.902759] Lustre: Mounted lustre-client - version 2.17.57_82_gfdb2c0c
[ 7318.371762] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7347.421357] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2