[ 1981.921905] LustreError: 26217:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 1986.046128] LustreError: lustre-OST0000-osc-ffff957608efe000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1991.975276] LustreError: 26217:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 awake
[ 1991.987874] Lustre: 2358:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.135@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1991.989183] LustreError: 26217:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 1992.175407] LustreError: 26217:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout interrupted
[ 1992.196204] Lustre: 2357:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.135@tcp:/lustre/fid: [0x200000404:0x36:0x0]// may get corrupted (rc -108)
[ 1994.617520] Lustre: DEBUG MARKER: recovery-small test_24b: @@@@@@ IGNORE (bz5494): multiop didn't fail fsync: 5 or close: 0
[ 2006.349950] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 14:34:54 (1790102094)
[ 2009.144266] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 2010.782407] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 14:34:59 (1790102099)
[ 2012.323495] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 2014.614977] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 14:35:02 (1790102102)
[ 2018.948710] LustreError: lustre-MDT0000-mdc-ffff957608efe000: operation mds_close to node 192.168.202.135@tcp failed: rc = -19
[ 2037.727255] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 192.168.202.135@tcp) was lost; in progress operations using this service will fail
[ 2047.911312] Lustre: Evicted from MGS (at 192.168.202.135@tcp) after server handle changed from 0xab723cec37171be3 to 0xab723cec371734e1
[ 2151.233063] LustreError: lustre-MDT0000-mdc-ffff957608efe000: operation mds_reint to node 192.168.202.135@tcp failed: rc = -19
[ 2151.247270] LustreError: Skipped 1 previous similar message
[ 2169.839357] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 192.168.202.135@tcp) was lost; in progress operations using this service will fail
[ 2169.862617] Lustre: Evicted from MGS (at 192.168.202.135@tcp) after server handle changed from 0xab723cec371734e1 to 0xab723cec37196765
[ 2182.782468] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 14:37:51 (1790102271)
[ 2183.073050] Lustre: *** cfs_fail_loc=305, val=0***
[ 2183.074830] Lustre: Skipped 2 previous similar messages
[ 2227.167401] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 192.168.202.135@tcp) was lost; in progress operations using this service will fail
[ 2227.191495] Lustre: Evicted from MGS (at 192.168.202.135@tcp) after server handle changed from 0xab723cec37196765 to 0xab723cec371995e8
[ 2228.725603] Lustre: 15926:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.135@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2236.397574] Lustre: lustre-MDT0000-mdc-ffff957608efe000: Connection to lustre-MDT0000 (at 192.168.202.135@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2236.414396] Lustre: Skipped 11 previous similar messages
[ 2236.486389] Lustre: lustre-MDT0000-mdc-ffff957608efe000: Connection restored to 192.168.202.135@tcp (at 192.168.202.135@tcp)
[ 2236.493528] Lustre: Skipped 14 previous similar messages
[ 2249.325809] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2250.684876] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2259.081207] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 14:39:07 (1790102347)
[ 2277.364080] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 192.168.202.135@tcp) was lost; in progress operations using this service will fail
[ 2277.390819] Lustre: Evicted from MGS (at 192.168.202.135@tcp) after server handle changed from 0xab723cec371995e8 to 0xab723cec37199e15
[ 2281.643496] LustreError: lustre-MDT0000-mdc-ffff957608efe000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2297.303392] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 14:39:45 (1790102385)
[ 2311.827509] LustreError: lustre-OST0000-osc-ffff957608efe000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2329.420481] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 14:40:17 (1790102417)
[ 2344.074121] LustreError: lustre-MDT0000-mdc-ffff957608efe000: operation ldlm_enqueue to node 192.168.202.135@tcp failed: rc = -19
[ 2344.082792] LustreError: Skipped 1 previous similar message
[ 2364.383210] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 192.168.202.135@tcp) was lost; in progress operations using this service will fail
[ 2374.588045] Lustre: Evicted from MGS (at 192.168.202.135@tcp) after server handle changed from 0xab723cec37199e15 to 0xab723cec3719eee9
[ 2393.675583] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2396.372081] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2482.145222] Lustre: 2357:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790102555/real 1790102555]  req@ffff9576057c1f80 x1877056129372800/t0(0) o400->MGC192.168.202.135@tcp@192.168.202.135@tcp:26/25 lens 224/224 e 0 to 1 dl 1790102571 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2482.182536] Lustre: 2357:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 16 previous similar messages
[ 2482.190643] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 192.168.202.135@tcp) was lost; in progress operations using this service will fail
[ 2491.391576] Lustre: Evicted from MGS (at 192.168.202.135@tcp) after server handle changed from 0xab723cec3719eee9 to 0xab723cec371ba859
[ 2509.038582] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2511.143088] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2597.865705] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 192.168.202.135@tcp) was lost; in progress operations using this service will fail
[ 2597.897282] Lustre: Evicted from MGS (at 192.168.202.135@tcp) after server handle changed from 0xab723cec371ba859 to 0xab723cec371d47c8
[ 2599.058635] Lustre: 15926:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.135@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2625.288546] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2628.200987] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2657.390880] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 14:45:45 (1790102745)
[ 2661.926157] LustreError: lustre-MDT0000-mdc-ffff957608efe000: operation mds_reint to node 192.168.202.135@tcp failed: rc = -19
[ 2661.944087] LustreError: Skipped 3 previous similar messages
[ 2679.775992] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 192.168.202.135@tcp) was lost; in progress operations using this service will fail
[ 2689.979347] Lustre: Evicted from MGS (at 192.168.202.135@tcp) after server handle changed from 0xab723cec371d47c8 to 0xab723cec371e13b6
[ 2692.091678] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2733.663158] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2780.046402] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2813.934904] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 192.168.202.135@tcp) was lost; in progress operations using this service will fail
[ 2813.946823] LustreError: Skipped 2 previous similar messages
[ 2825.057680] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2848.673575] Lustre: lustre-MDT0000-mdc-ffff957608efe000: Connection to lustre-MDT0000 (at 192.168.202.135@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2848.691733] Lustre: Skipped 7 previous similar messages
[ 2876.930136] Lustre: Evicted from MGS (at 192.168.202.135@tcp) after server handle changed from 0xab723cec371e35c1 to 0xab723cec371f0505
[ 2876.952684] Lustre: Skipped 3 previous similar messages
[ 2876.970253] Lustre: MGC192.168.202.135@tcp: Connection restored to 192.168.202.135@tcp (at 192.168.202.135@tcp)
[ 2876.986633] Lustre: Skipped 15 previous similar messages
[ 2886.569144] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2958.481608] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 3024.042055] Lustre: 15926:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.135@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 3062.220232] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 14:52:30 (1790103150)
[ 3118.383962] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3120.801099] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3407.372189] LustreError: lustre-OST0000-osc-ffff957608efe000: operation ldlm_enqueue to node 192.168.202.135@tcp failed: rc = -107
[ 3407.380506] LustreError: Skipped 7 previous similar messages
[ 3454.423737] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3456.875272] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3738.294828] Lustre: lustre-OST0000-osc-ffff957608efe000: Connection to lustre-OST0000 (at 192.168.202.135@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3738.306415] Lustre: Skipped 4 previous similar messages
[ 3771.271091] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3773.057888] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4035.653964] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 15:08:43 (1790104123)
[ 4053.983169] Lustre: 50852:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790104127/real 1790104127]  req@ffff9576057b6300 x1877056153030784/t0(0) o101->lustre-MDT0000-mdc-ffff957608efe000@192.168.202.135@tcp:12/10 lens 576/1152 e 0 to 1 dl 1790104143 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 4054.014048] Lustre: 50852:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
[ 4054.064972] Lustre: lustre-MDT0000-mdc-ffff957608efe000: Connection restored to 192.168.202.135@tcp (at 192.168.202.135@tcp)
[ 4054.076905] Lustre: Skipped 6 previous similar messages
[ 4061.947888] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 15:09:10 (1790104150)
[ 4090.559699] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 15:09:38 (1790104178)
[ 4118.539988] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 15:10:06 (1790104206)
[ 4119.208157] Lustre: Mounted lustre-client - version 2.17.58_63_g8caac7f
[ 4150.304871] Lustre: 2358:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790104224/real 1790104224]  req@ffff9576087cf800 x1877056153049472/t0(0) o400->lustre-MDT0000-mdc-ffff957608efe000@192.168.202.135@tcp:12/10 lens 224/224 e 0 to 1 dl 1790104240 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4150.333597] Lustre: 2358:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 4150.347926] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 192.168.202.135@tcp) was lost; in progress operations using this service will fail
[ 4150.358256] LustreError: Skipped 3 previous similar messages
[ 4161.011414] Lustre: Evicted from MGS (at 192.168.202.135@tcp) after server handle changed from 0xab723cec372034f2 to 0xab723cec37377116
[ 4161.027388] Lustre: Skipped 2 previous similar messages
[ 4167.373522] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4169.245989] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4171.861030] Lustre: Unmounted lustre-client
[ 4180.610187] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 15:11:08 (1790104268)
[ 4305.375150] Lustre: 2357:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790104379/real 1790104379]  req@ffff9576057b5c00 x1877056153083008/t0(0) o4->lustre-OST0000-osc-ffff957608efe000@192.168.202.135@tcp:6/4 lens 488/448 e 0 to 1 dl 1790104395 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4305.405400] Lustre: 2357:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 51 previous similar messages
[ 4353.439267] Lustre: lustre-OST0000-osc-ffff957608efe000: Connection to lustre-OST0000 (at 192.168.202.135@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4353.448901] Lustre: Skipped 14 previous similar messages
[ 4478.727672] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 15:16:07 (1790104567)
[ 4526.913194] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 15:16:55 (1790104615)
[ 4529.652284] Lustre: Unmounted lustre-client
[ 4551.275460] Lustre: Mounted lustre-client - version 2.17.58_63_g8caac7f
[ 4558.272680] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 15:17:26 (1790104646)
[ 4558.538093] LustreError: 56782:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4559.589530] LustreError: 56782:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4559.761270] Lustre: *** cfs_fail_loc=305, val=0***
[ 4583.366794] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 15:17:51 (1790104671)
[ 4583.783408] Lustre: Mounted lustre-client - version 2.17.58_63_g8caac7f
[ 4586.141930] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4596.448554] Lustre: Unmounted lustre-client
[ 4604.194541] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 15:18:12 (1790104692)
[ 4689.936626] LustreError: lustre-MDT0000-mdc-ffff957608ec0000: operation mds_reint to node 192.168.202.135@tcp failed: rc = -19
[ 4689.942600] LustreError: Skipped 1 previous similar message
[ 4707.615101] Lustre: 2357:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790104781/real 1790104781]  req@ffff9576091f1880 x1877056155746944/t0(0) o400->MGC192.168.202.135@tcp@192.168.202.135@tcp:26/25 lens 224/224 e 0 to 1 dl 1790104797 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4707.673894] Lustre: 2357:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 66 previous similar messages
[ 4707.688163] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 192.168.202.135@tcp) was lost; in progress operations using this service will fail
[ 4717.883558] Lustre: Evicted from MGS (at 192.168.202.135@tcp) after server handle changed from 0xab723cec37378b02 to 0xab723cec373a3995
[ 4717.920582] Lustre: MGC192.168.202.135@tcp: Connection restored to 192.168.202.135@tcp (at 192.168.202.135@tcp)
[ 4717.937515] Lustre: Skipped 23 previous similar messages
[ 4719.850451] Lustre: 56180:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.135@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 4873.445373] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 15:22:41 (1790104961)
[ 4879.035714] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4896.238157] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 192.168.202.135@tcp) was lost; in progress operations using this service will fail
[ 4896.259498] Lustre: Evicted from MGS (at 192.168.202.135@tcp) after server handle changed from 0xab723cec373a3995 to 0xab723cec373f0f14
[ 4896.325589] LustreError: lustre-MDT0000-mdc-ffff957608ec0000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4908.534795] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 15:23:16 (1790104996)
[ 4908.913327] Lustre: Mounted lustre-client - version 2.17.58_63_g8caac7f
[ 4916.339927] LustreError: lustre-OST0000-osc-ffff957608ec0000: operation ldlm_enqueue to node 192.168.202.135@tcp failed: rc = -107
[ 4916.359373] LustreError: lustre-OST0000-osc-ffff957608ec0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4927.614182] Lustre: Unmounted lustre-client
[ 4933.617326] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 15:23:42 (1790105022)
[ 4940.225504] LustreError: lustre-MDT0000-mdc-ffff957608ec0000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4940.244836] LustreError: 61177:0:(file.c:6224:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4948.853557] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 15:23:57 (1790105037)
[ 4949.340536] LustreError: 61881:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4954.591152] LustreError: 61881:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=0
[ 4954.601736] LustreError: 61881:0:(import.c:756:ptlrpc_connect_import_locked()) already connected
[ 4955.192441] Lustre: lustre-MDT0000-mdc-ffff957608ec0000: Connection to lustre-MDT0000 (at 192.168.202.135@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4955.218442] Lustre: Skipped 17 previous similar messages
[ 4955.245178] LustreError: lustre-MDT0000-mdc-ffff957608ec0000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4955.261068] LustreError: 61899:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4955.361538] LustreError: 61904:0:(file.c:6224:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4956.490109] LustreError: 61910:0:(file.c:6224:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4956.503477] LustreError: 61910:0:(file.c:6224:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4958.728664] LustreError: 61932:0:(file.c:6224:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4958.752187] LustreError: 61932:0:(file.c:6224:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4963.418606] LustreError: 61977:0:(file.c:6224:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4963.429328] LustreError: 61977:0:(file.c:6224:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4972.835427] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 15:24:21 (1790105061)
[ 5010.887561] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5012.342528] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5023.231772] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 15:25:11 (1790105111)
[ 5069.720886] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5071.541927] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5084.666662] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 15:26:12 (1790105172)
[ 5145.861493] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5147.513992] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5156.006373] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 15:27:24 (1790105244)
[ 5198.446971] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5200.450361] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5207.221293] Lustre: Unmounted lustre-client
[ 5256.678631] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5259.171479] Lustre: Mounted lustre-client - version 2.17.58_63_g8caac7f
[ 5265.119163] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 15:29:13 (1790105353)
[ 5267.242889] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5268.909674] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 15:29:17 (1790105357)
[ 5298.037551] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 15:29:46 (1790105386)
[ 5300.054801] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5302.174190] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 15:29:50 (1790105390)
[ 5304.203725] Lustre: *** cfs_fail_loc=805, val=0***
[ 5304.277133] Lustre: Mounted lustre-client - version 2.17.58_63_g8caac7f
[ 5310.426897] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5330.911342] Lustre: 2359:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790105404/real 1790105404]  req@ffff95760906ed80 x1877056157813888/t0(0) o400->lustre-MDT0000-mdc-ffff9576035d3800@192.168.202.135@tcp:12/10 lens 224/224 e 0 to 1 dl 1790105420 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5330.969662] Lustre: 2359:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5330.978146] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 192.168.202.135@tcp) was lost; in progress operations using this service will fail
[ 5341.170281] LustreError: 2356:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9576093c3000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff957609c31d00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5341.192730] Lustre: lustre-MDT0000-mdc-ffff9576093c3000: Connection restored to 192.168.202.135@tcp (at 192.168.202.135@tcp)
[ 5341.207867] Lustre: Skipped 12 previous similar messages
[ 5341.218763] Lustre: Evicted from MGS (at 192.168.202.135@tcp) after server handle changed from 0xab723cec373f1939 to 0xab723cec373f1cea
[ 5343.702214] Lustre: Unmounted lustre-client
[ 5351.659600] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 15:30:39 (1790105439)
[ 5394.378841] LustreError: 2356:0:(client.c:3449:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff957608636d80 x1877056157825664/t85899345924(85899345924) o101->lustre-MDT0000-mdc-ffff9576035d3800@192.168.202.135@tcp:12/10 lens 648/608 e 0 to 0 dl 1790105500 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5400.869212] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5402.920042] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5411.024095] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 15:31:39 (1790105499)
[ 5412.490126] LustreError: lustre-OST0000-osc-ffff9576035d3800: operation ost_write to node 192.168.202.135@tcp failed: rc = -107
[ 5412.519084] LustreError: Skipped 2 previous similar messages
[ 5412.555384] LustreError: lustre-OST0000-osc-ffff9576035d3800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5412.568713] Lustre: 2359:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.135@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5412.623484] LustreError: 73081:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff9576035d3800: namespace resource [0x240000400:0x4282:0x0].0x0 (ffff957609c2ab00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5412.653254] LustreError: 73081:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5421.193792] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 15:31:49 (1790105509)
[ 5422.750692] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5424.402689] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 15:31:52 (1790105512)
[ 5425.740954] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5427.291690] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 15:31:55 (1790105515)
[ 5428.828459] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5430.779948] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 15:31:58 (1790105518)
[ 5432.420973] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5434.519970] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 15:32:02 (1790105522)
[ 5436.155877] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5437.993229] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 15:32:06 (1790105526)
[ 5440.145988] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5442.243583] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 15:32:10 (1790105530)
[ 5443.859328] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5445.707362] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 15:32:13 (1790105533)
[ 5447.449069] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5449.435967] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 15:32:17 (1790105537)
[ 5450.997730] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5452.662518] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 15:32:21 (1790105541)
[ 5454.620973] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5456.296947] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 15:32:24 (1790105544)
[ 5458.014231] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5460.020852] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 15:32:28 (1790105548)
[ 5462.672960] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5464.939818] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 15:32:33 (1790105553)
[ 5500.615251] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 15:33:08 (1790105588)
[ 5531.046807] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 15:33:39 (1790105619)
[ 5531.397243] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5541.473498] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 15:33:49 (1790105629)
[ 5541.792886] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5553.831545] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 15:34:02 (1790105642)
[ 5554.509983] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5563.718947] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 15:34:12 (1790105652)
[ 5564.052592] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5572.419607] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 15:34:20 (1790105660)
[ 5572.725994] Lustre: *** cfs_fail_loc=510, val=3***
[ 5580.793766] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 15:34:29 (1790105669)
[ 5581.336715] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5591.976485] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 15:34:40 (1790105680)
[ 5592.346612] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5652.447497] Lustre: lustre-OST0001-osc-ffff9576035d3800: Connection to lustre-OST0001 (at 192.168.202.135@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5652.471519] Lustre: Skipped 9 previous similar messages
[ 5661.524713] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 15:35:49 (1790105749)
[ 5661.868961] LustreError: 83292:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5665.120633] LustreError: lustre-MDT0000-mdc-ffff9576035d3800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5665.161714] LustreError: 83306:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9576035d3800: inode [0x20000a042:0x10:0x0] mdc close failed: rc = -108
[ 5665.209260] LustreError: 83306:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9576035d3800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff957608efa900) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5665.951141] LustreError: 83292:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5668.063399] LustreError: 83322:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5670.962446] LustreError: lustre-MDT0000-mdc-ffff9576035d3800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5670.985544] LustreError: 83336:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9576035d3800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff95760a68d700) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5671.026755] LustreError: 83336:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5672.143127] LustreError: 83322:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5679.511813] LustreError: lustre-MDT0000-mdc-ffff9576035d3800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5679.962471] LustreError: 83377:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5682.656471] LustreError: lustre-MDT0000-mdc-ffff9576035d3800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5682.731826] LustreError: 83397:0:(file.c:6224:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5682.738572] LustreError: 83397:0:(file.c:6224:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5683.837767] LustreError: 83403:0:(file.c:6224:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5683.845256] LustreError: 83403:0:(file.c:6224:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5684.023233] LustreError: 83377:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5684.041663] LustreError: 83377:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9576035d3800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5686.195149] LustreError: 83426:0:(file.c:6224:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5686.217088] LustreError: 83426:0:(file.c:6224:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5689.977725] LustreError: 83452:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5692.866282] LustreError: lustre-MDT0000-mdc-ffff9576035d3800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5692.891828] LustreError: 83467:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9576035d3800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff95760a68d600) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5692.931357] LustreError: 83467:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5694.056064] LustreError: 83452:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5700.883346] LustreError: lustre-MDT0000-mdc-ffff9576035d3800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5701.190187] LustreError: 83509:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5703.992134] LustreError: lustre-MDT0000-mdc-ffff9576035d3800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5704.018358] LustreError: 83523:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9576035d3800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff957609c31200) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5704.032294] LustreError: 83523:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5705.256266] LustreError: 83509:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5708.567970] LustreError: lustre-MDT0000-mdc-ffff9576035d3800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5708.633532] LustreError: 83556:0:(file.c:6224:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5708.654479] LustreError: 83556:0:(file.c:6224:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5709.488616] LustreError: 83536:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9576035d3800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5717.824565] LustreError: 83608:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5717.837762] LustreError: 83608:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5721.729390] LustreError: lustre-MDT0000-mdc-ffff9576035d3800: operation ldlm_enqueue to node 192.168.202.135@tcp failed: rc = -107
[ 5721.755859] LustreError: Skipped 8 previous similar messages
[ 5721.780421] LustreError: lustre-MDT0000-mdc-ffff9576035d3800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5721.849283] LustreError: 83631:0:(file.c:6224:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5721.855406] LustreError: 83608:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5721.855417] LustreError: 83608:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5721.855955] LustreError: 83608:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9576035d3800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5721.857883] LustreError: 83631:0:(file.c:6224:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5734.632687] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 15:37:02 (1790105822)
[ 5752.826787] LustreError: 68538:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5752.834360] LustreError: 68538:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5756.903353] LustreError: 68538:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5756.921365] LustreError: 68538:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5763.198539] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 15:37:31 (1790105851)
[ 5813.305798] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 15:38:21 (1790105901)
[ 5879.202476] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 15:39:27 (1790105967)
[ 5915.530816] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 15:40:03 (1790106003)
[ 5916.135430] Lustre: Mounted lustre-client - version 2.17.58_63_g8caac7f
[ 6040.949663] Lustre: Unmounted lustre-client
[ 6047.474979] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 15:42:15 (1790106135)
[ 6048.068150] LustreError: 87933:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 6050.565906] LustreError: lustre-OST0000-osc-ffff9576035d3800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6050.594265] LustreError: 88015:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff9576035d3800: namespace resource [0x240000400:0x42ad:0x0].0x0 (ffff95760a68dc00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6050.620557] LustreError: 88015:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 6051.112879] LustreError: 87933:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout interrupted
[ 6051.124930] Lustre: 2358:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.135@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 6058.178739] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 15:42:26 (1790106146)
[ 6117.344603] Lustre: 88600:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790106151/real 1790106151]  req@ffff95760906e680 x1877056158000128/t0(0) o101->lustre-MDT0000-mdc-ffff9576035d3800@192.168.202.135@tcp:12/10 lens 328/344 e 0 to 1 dl 1790106207 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 6117.369269] Lustre: 88600:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 20 previous similar messages
[ 6117.386641] Lustre: lustre-MDT0000-mdc-ffff9576035d3800: Connection restored to 192.168.202.135@tcp (at 192.168.202.135@tcp)
[ 6117.390959] Lustre: Skipped 18 previous similar messages
[ 6126.763950] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 15:43:34 (1790106214)
[ 6128.579332] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 6130.870950] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 15:43:38 (1790106218)
[ 6196.344528] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6197.851692] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 15:44:46 (1790106286)
[ 6254.559228] Lustre: lustre-MDT0000-mdc-ffff9576035d3800: Connection to lustre-MDT0000 (at 192.168.202.135@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6254.574920] Lustre: Skipped 15 previous similar messages
[ 6256.105372] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 15:45:44 (1790106344)
[ 6258.451609] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6259.871159] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 15:45:48 (1790106348)
[ 6261.231204] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6262.696313] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 15:45:51 (1790106351)
[ 6289.742287] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 15:46:18 (1790106378)
[ 6300.566756] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6327.199222] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 192.168.202.135@tcp) was lost; in progress operations using this service will fail
[ 6327.213324] LustreError: Skipped 2 previous similar messages
[ 6336.362840] Lustre: Evicted from MGS (at 192.168.202.135@tcp) after server handle changed from 0xab723cec373f2771 to 0xab723cec373f4099
[ 6336.372557] Lustre: Skipped 2 previous similar messages
[ 6349.118333] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6350.883722] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6359.753505] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 15:47:28 (1790106448)
[ 6361.564388] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6363.512300] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 15:47:31 (1790106451)
[ 6374.374885] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 192.168.202.135@tcp) was lost; in progress operations using this service will fail
[ 6374.423634] Lustre: Evicted from MGS (at 192.168.202.135@tcp) after server handle changed from 0xab723cec373f4099 to 0xab723cec373f43d3
[ 6385.877327] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 15:47:54 (1790106474)
[ 6428.820634] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 15:48:37 (1790106517)
[ 6434.832098] LustreError: lustre-OST0000-osc-ffff9576035d3800: operation ost_setattr to node 192.168.202.135@tcp failed: rc = -107
[ 6434.842049] LustreError: Skipped 1 previous similar message
[ 6472.859364] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6474.784104] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6558.698764] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 192.168.202.135@tcp) was lost; in progress operations using this service will fail
[ 6558.715137] LustreError: Skipped 1 previous similar message
[ 6558.736752] Lustre: Evicted from MGS (at 192.168.202.135@tcp) after server handle changed from 0xab723cec373f475a to 0xab723cec373f5625
[ 6558.761992] Lustre: Skipped 1 previous similar message
[ 6568.644276] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6570.098847] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6601.904768] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6603.058782] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6692.060909] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 15:53:00 (1790106780)
[ 6739.970102] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6743.046277] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6951.932367] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 15:57:20 (1790107040)
[ 7060.453922] Lustre: lustre-MDT0000-mdc-ffff9576035d3800: Connection to lustre-MDT0000 (at 192.168.202.135@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7060.466752] Lustre: Skipped 7 previous similar messages
[ 7060.470771] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 192.168.202.135@tcp) was lost; in progress operations using this service will fail
[ 7060.477515] LustreError: Skipped 1 previous similar message
[ 7060.504814] Lustre: Evicted from MGS (at 192.168.202.135@tcp) after server handle changed from 0xab723cec373f5c14 to 0xab723cec374efca9
[ 7060.510800] Lustre: Skipped 1 previous similar message
[ 7060.531159] Lustre: MGC192.168.202.135@tcp: Connection restored to 192.168.202.135@tcp (at 192.168.202.135@tcp)
[ 7060.540450] Lustre: Skipped 12 previous similar messages
[ 7083.210362] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 15:59:31 (1790107171)
[ 7084.204959] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7085.356750] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 15:59:34 (1790107174)
[ 7086.721331] LustreError: lustre-MDT0000-mdc-ffff9576035d3800: operation ldlm_enqueue to node 192.168.202.135@tcp failed: rc = -107
[ 7086.734852] LustreError: Skipped 113 previous similar messages
[ 7086.762731] LustreError: lustre-MDT0000-mdc-ffff9576035d3800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7091.848193] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 15:59:40 (1790107180)
[ 7259.116450] LustreError: lustre-OST0000-osc-ffff9576035d3800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7260.126417] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 16:02:29 (1790107349)
[ 7283.295217] Lustre: 2358:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790107353/real 1790107353]  req@ffff9576255b8000 x1877056175704320/t0(0) o4->lustre-OST0000-osc-ffff9576035d3800@192.168.202.135@tcp:6/4 lens 4584/448 e 0 to 1 dl 1790107373 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7283.339781] Lustre: 2358:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 24 previous similar messages
[ 7300.440926] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 16:03:08 (1790107388)
[ 7301.468727] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7302.789860] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 16:03:11 (1790107391)
[ 7304.413376] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7306.190931] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 16:03:14 (1790107394)
[ 7348.165700] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 16:03:54 (1790107434)
[ 7392.241222] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 192.168.202.135@tcp) was lost; in progress operations using this service will fail
[ 7392.274848] Lustre: Evicted from MGS (at 192.168.202.135@tcp) after server handle changed from 0xab723cec374efca9 to 0xab723cec374f24b8
[ 7392.297447] Lustre: Skipped 1 previous similar message
[ 7413.001359] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 16:05:00 (1790107500)
[ 7415.709564] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7417.743691] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 16:05:05 (1790107505)
[ 7419.913928] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7422.401197] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 16:05:10 (1790107510)
[ 7434.155476] Lustre: Unmounted lustre-client
[ 7435.880265] Lustre: Mounted lustre-client - version 2.17.58_63_g8caac7f
[ 7439.035189] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7463.880384] Lustre: 107040:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.135@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 7483.905739] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2