[ 1968.617631] LustreError: 34813:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 1970.663500] LustreError: 37:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 1972.743698] LustreError: lustre-OST0000-osc-ffff8d5e98e8a000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1978.639243] LustreError: 34813:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 awake
[ 1978.645826] Lustre: 2354:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.107@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1978.791094] LustreError: 37:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout interrupted
[ 1978.805728] Lustre: 2352:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.107@tcp:/lustre/fid: [0x200000404:0x36:0x0]// may get corrupted (rc -108)
[ 1980.918339] Lustre: DEBUG MARKER: recovery-small test_24b: @@@@@@ IGNORE (bz5494): multiop didn't fail fsync: 108 or close: 0
[ 1990.606693] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 19:32:53 (1790897573)
[ 1993.282792] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1994.912560] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 19:32:57 (1790897577)
[ 1996.381362] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1998.332378] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 19:33:01 (1790897581)
[ 2003.041909] LustreError: lustre-MDT0000-mdc-ffff8d5e98e8a000: operation ldlm_enqueue to node 192.168.204.107@tcp failed: rc = -19
[ 2023.904064] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 192.168.204.107@tcp) was lost; in progress operations using this service will fail
[ 2034.228433] Lustre: Evicted from MGS (at 192.168.204.107@tcp) after server handle changed from 0x5676da2a5e795ec3 to 0x5676da2a5e797671
[ 2141.610271] LustreError: lustre-MDT0000-mdc-ffff8d5e98e8a000: operation ldlm_enqueue to node 192.168.204.107@tcp failed: rc = -19
[ 2141.633858] LustreError: Skipped 1 previous similar message
[ 2161.142493] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 192.168.204.107@tcp) was lost; in progress operations using this service will fail
[ 2161.190166] Lustre: Evicted from MGS (at 192.168.204.107@tcp) after server handle changed from 0x5676da2a5e797671 to 0x5676da2a5e7afb22
[ 2193.222391] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 19:36:15 (1790897775)
[ 2193.960265] Lustre: *** cfs_fail_loc=305, val=0***
[ 2193.963360] Lustre: Skipped 4 previous similar messages
[ 2240.479973] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 192.168.204.107@tcp) was lost; in progress operations using this service will fail
[ 2250.757201] Lustre: Evicted from MGS (at 192.168.204.107@tcp) after server handle changed from 0x5676da2a5e7afb22 to 0x5676da2a5e7b01dc
[ 2269.247522] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2271.731909] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2280.945906] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 19:37:43 (1790897863)
[ 2296.812925] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 192.168.204.107@tcp) was lost; in progress operations using this service will fail
[ 2296.864821] Lustre: Evicted from MGS (at 192.168.204.107@tcp) after server handle changed from 0x5676da2a5e7b01dc to 0x5676da2a5e7b0682
[ 2296.913928] LustreError: lustre-MDT0000-mdc-ffff8d5e98e8a000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2326.318617] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 19:38:28 (1790897908)
[ 2342.986087] LustreError: lustre-OST0000-osc-ffff8d5e98e8a000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2365.458871] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 19:39:08 (1790897948)
[ 2379.242665] LustreError: lustre-MDT0000-mdc-ffff8d5e98e8a000: operation mds_reint to node 192.168.204.107@tcp failed: rc = -19
[ 2379.257063] LustreError: Skipped 2 previous similar messages
[ 2400.224633] Lustre: 2354:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790897968/real 1790897968]  req@ffff8d5e87f94000 x1877890189650560/t0(0) o400->MGC192.168.204.107@tcp@192.168.204.107@tcp:26/25 lens 224/224 e 0 to 1 dl 1790897984 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2400.268413] Lustre: 2354:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 17 previous similar messages
[ 2400.286802] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 192.168.204.107@tcp) was lost; in progress operations using this service will fail
[ 2410.493692] Lustre: Evicted from MGS (at 192.168.204.107@tcp) after server handle changed from 0x5676da2a5e7b0682 to 0x5676da2a5e7b4f99
[ 2429.162183] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2432.117085] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2516.959246] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 192.168.204.107@tcp) was lost; in progress operations using this service will fail
[ 2527.151215] Lustre: Evicted from MGS (at 192.168.204.107@tcp) after server handle changed from 0x5676da2a5e7b4f99 to 0x5676da2a5e7cad14
[ 2527.175966] Lustre: MGC192.168.204.107@tcp: Connection restored to 192.168.204.107@tcp (at 192.168.204.107@tcp)
[ 2527.185230] Lustre: Skipped 13 previous similar messages
[ 2545.303576] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2548.068789] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2614.603048] Lustre: lustre-MDT0000-mdc-ffff8d5e98e8a000: Connection to lustre-MDT0000 (at 192.168.204.107@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2614.618139] Lustre: Skipped 9 previous similar messages
[ 2633.752597] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 192.168.204.107@tcp) was lost; in progress operations using this service will fail
[ 2643.955606] Lustre: Evicted from MGS (at 192.168.204.107@tcp) after server handle changed from 0x5676da2a5e7cad14 to 0x5676da2a5e7e3f1d
[ 2663.323400] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2666.950653] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2698.646164] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 19:44:41 (1790898281)
[ 2703.830809] LustreError: lustre-MDT0000-mdc-ffff8d5e98e8a000: operation ldlm_enqueue to node 192.168.204.107@tcp failed: rc = -19
[ 2703.851034] LustreError: Skipped 2 previous similar messages
[ 2720.736359] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 192.168.204.107@tcp) was lost; in progress operations using this service will fail
[ 2730.999852] Lustre: Evicted from MGS (at 192.168.204.107@tcp) after server handle changed from 0x5676da2a5e7e3f1d to 0x5676da2a5e7ef0fc
[ 2732.412905] Lustre: 15963:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.107@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2747.114975] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2792.260459] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2839.485893] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2871.263313] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 192.168.204.107@tcp) was lost; in progress operations using this service will fail
[ 2871.286786] LustreError: Skipped 2 previous similar messages
[ 2881.544447] Lustre: Evicted from MGS (at 192.168.204.107@tcp) after server handle changed from 0x5676da2a5e7f62f4 to 0x5676da2a5e7fa796
[ 2881.561907] Lustre: Skipped 2 previous similar messages
[ 2894.394266] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2947.070741] Lustre: 15963:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.107@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2959.424857] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 3007.455136] Lustre: 2351:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790898575/real 1790898575]  req@ffff8d5e8829ed80 x1877890194373760/t0(0) o400->MGC192.168.204.107@tcp@192.168.204.107@tcp:26/25 lens 224/224 e 0 to 1 dl 1790898591 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3007.497021] Lustre: 2351:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
[ 3027.694905] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 3128.443831] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 19:51:50 (1790898710)
[ 3167.651299] Lustre: lustre-OST0000-osc-ffff8d5e98e8a000: Connection restored to 192.168.204.107@tcp (at 192.168.204.107@tcp)
[ 3167.660203] Lustre: Skipped 17 previous similar messages
[ 3192.492699] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3194.917181] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3473.414809] LustreError: lustre-OST0000-osc-ffff8d5e98e8a000: operation ldlm_enqueue to node 192.168.204.107@tcp failed: rc = -19
[ 3473.429904] LustreError: Skipped 10 previous similar messages
[ 3473.436949] Lustre: lustre-OST0000-osc-ffff8d5e98e8a000: Connection to lustre-OST0000 (at 192.168.204.107@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3473.463953] Lustre: Skipped 8 previous similar messages
[ 3519.091291] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3521.596788] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3827.835229] Lustre: lustre-OST0000-osc-ffff8d5e98e8a000: Connection restored to 192.168.204.107@tcp (at 192.168.204.107@tcp)
[ 3827.856996] Lustre: Skipped 1 previous similar message
[ 3847.195703] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3849.739474] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4101.073983] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 20:08:03 (1790899683)
[ 4117.983885] Lustre: 50890:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790899686/real 1790899686]  req@ffff8d5e900dbb80 x1877890211330688/t0(0) o101->lustre-MDT0000-mdc-ffff8d5e98e8a000@192.168.204.107@tcp:12/10 lens 576/1152 e 0 to 1 dl 1790899702 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 4118.025649] Lustre: 50890:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 4118.038173] Lustre: lustre-MDT0000-mdc-ffff8d5e98e8a000: Connection to lustre-MDT0000 (at 192.168.204.107@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4118.058352] Lustre: Skipped 1 previous similar message
[ 4125.938125] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 20:08:28 (1790899708)
[ 4150.588297] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 20:08:53 (1790899733)
[ 4176.171877] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 20:09:19 (1790899759)
[ 4176.587369] Lustre: Mounted lustre-client - version 2.17.58_63_g3705aed
[ 4208.415377] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 192.168.204.107@tcp) was lost; in progress operations using this service will fail
[ 4208.434090] LustreError: Skipped 3 previous similar messages
[ 4217.843665] Lustre: Evicted from MGS (at 192.168.204.107@tcp) after server handle changed from 0x5676da2a5e818fc8 to 0x5676da2a5e94bd4c
[ 4217.859568] Lustre: Skipped 3 previous similar messages
[ 4224.318401] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4226.099497] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4228.153664] Lustre: Unmounted lustre-client
[ 4234.860633] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 20:10:17 (1790899817)
[ 4437.994353] Lustre: lustre-OST0000-osc-ffff8d5e98e8a000: Connection restored to 192.168.204.107@tcp (at 192.168.204.107@tcp)
[ 4438.005806] Lustre: Skipped 17 previous similar messages
[ 4531.618418] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 20:15:14 (1790900114)
[ 4579.873536] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 20:16:02 (1790900162)
[ 4583.018373] Lustre: Unmounted lustre-client
[ 4605.353088] Lustre: Mounted lustre-client - version 2.17.58_63_g3705aed
[ 4612.421838] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 20:16:34 (1790900194)
[ 4612.633833] LustreError: 56811:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4613.688617] LustreError: 56811:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4613.862718] Lustre: *** cfs_fail_loc=305, val=0***
[ 4638.414221] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 20:17:01 (1790900221)
[ 4638.807310] Lustre: Mounted lustre-client - version 2.17.58_63_g3705aed
[ 4640.380940] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4650.725322] Lustre: Unmounted lustre-client
[ 4657.386371] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 20:17:20 (1790900240)
[ 4740.173297] LustreError: lustre-MDT0000-mdc-ffff8d5e86f6d800: operation mds_reint to node 192.168.204.107@tcp failed: rc = -19
[ 4740.188097] LustreError: Skipped 1 previous similar message
[ 4740.198633] Lustre: lustre-MDT0000-mdc-ffff8d5e86f6d800: Connection to lustre-MDT0000 (at 192.168.204.107@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4740.213625] Lustre: Skipped 23 previous similar messages
[ 4758.495260] Lustre: 2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790900326/real 1790900326]  req@ffff8d5e912cb800 x1877890214050944/t0(0) o400->MGC192.168.204.107@tcp@192.168.204.107@tcp:26/25 lens 224/224 e 0 to 1 dl 1790900342 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4758.530861] Lustre: 2352:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 143 previous similar messages
[ 4758.548752] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 192.168.204.107@tcp) was lost; in progress operations using this service will fail
[ 4768.757274] Lustre: Evicted from MGS (at 192.168.204.107@tcp) after server handle changed from 0x5676da2a5e94d3b1 to 0x5676da2a5e9787f4
[ 4921.403266] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 20:21:44 (1790900504)
[ 4926.681100] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4940.777542] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 192.168.204.107@tcp) was lost; in progress operations using this service will fail
[ 4940.816292] Lustre: Evicted from MGS (at 192.168.204.107@tcp) after server handle changed from 0x5676da2a5e9787f4 to 0x5676da2a5e9c5213
[ 4940.888274] LustreError: lustre-MDT0000-mdc-ffff8d5e86f6d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4957.367556] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 20:22:20 (1790900540)
[ 4957.766778] Lustre: Mounted lustre-client - version 2.17.58_63_g3705aed
[ 4965.241951] LustreError: lustre-OST0000-osc-ffff8d5e86f6d800: operation ldlm_enqueue to node 192.168.204.107@tcp failed: rc = -107
[ 4965.259438] LustreError: lustre-OST0000-osc-ffff8d5e86f6d800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4977.461033] Lustre: Unmounted lustre-client
[ 4984.679816] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 20:22:47 (1790900567)
[ 4992.090488] LustreError: lustre-MDT0000-mdc-ffff8d5e86f6d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4992.140115] LustreError: 61201:0:(file.c:6224:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 5000.755219] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 20:23:03 (1790900583)
[ 5001.013494] LustreError: 61907:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 5005.620129] LustreError: lustre-MDT0000-mdc-ffff8d5e86f6d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5005.637199] LustreError: 61924:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 5005.653891] LustreError: 61907:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=367
[ 5005.678704] LustreError: 61907:0:(import.c:762:ptlrpc_connect_import_locked()) already connecting
[ 5005.695817] LustreError: 61929:0:(file.c:6224:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5006.771788] LustreError: 61935:0:(file.c:6224:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5006.791064] LustreError: 61935:0:(file.c:6224:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5009.070072] LustreError: 61958:0:(file.c:6224:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5009.095224] LustreError: 61958:0:(file.c:6224:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5013.692145] LustreError: 62003:0:(file.c:6224:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5013.715452] LustreError: 62003:0:(file.c:6224:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 5024.418519] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 20:23:27 (1790900607)
[ 5065.273498] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5067.167745] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5077.348223] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 20:24:20 (1790900660)
[ 5100.727267] Lustre: lustre-OST0000-osc-ffff8d5e86f6d800: Connection restored to 192.168.204.107@tcp (at 192.168.204.107@tcp)
[ 5100.738645] Lustre: Skipped 17 previous similar messages
[ 5113.461553] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5115.211525] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5124.271801] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 20:25:07 (1790900707)
[ 5187.548852] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5189.308615] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5197.524732] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 20:26:20 (1790900780)
[ 5239.668988] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5241.115695] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5246.465817] Lustre: Unmounted lustre-client
[ 5302.411221] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5305.246416] Lustre: Mounted lustre-client - version 2.17.58_63_g3705aed
[ 5312.382109] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 20:28:15 (1790900895)
[ 5315.309358] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5317.030359] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 20:28:19 (1790900899)
[ 5349.194766] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 20:28:52 (1790900932)
[ 5350.759749] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5352.382740] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 20:28:55 (1790900935)
[ 5353.560406] Lustre: *** cfs_fail_loc=805, val=0***
[ 5353.613606] Lustre: Mounted lustre-client - version 2.17.58_63_g3705aed
[ 5358.775807] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5380.511919] Lustre: 2351:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790900948/real 1790900948]  req@ffff8d5e88286d80 x1877890216099072/t0(0) o400->lustre-MDT0000-mdc-ffff8d5e842e3800@192.168.204.107@tcp:12/10 lens 224/224 e 0 to 1 dl 1790900964 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5380.575252] Lustre: lustre-MDT0000-mdc-ffff8d5e98c8a000: Connection to lustre-MDT0000 (at 192.168.204.107@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5380.585914] Lustre: 2351:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 5380.640042] Lustre: Skipped 13 previous similar messages
[ 5380.644777] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 192.168.204.107@tcp) was lost; in progress operations using this service will fail
[ 5389.804361] Lustre: Evicted from MGS (at 192.168.204.107@tcp) after server handle changed from 0x5676da2a5e9c5c07 to 0x5676da2a5e9c5efb
[ 5401.186764] Lustre: Unmounted lustre-client
[ 5409.868949] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 20:29:52 (1790900992)
[ 5455.371659] LustreError: 2350:0:(client.c:3449:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff8d5e9128dc00 x1877890216111616/t94489280516(94489280516) o101->lustre-MDT0000-mdc-ffff8d5e842e3800@192.168.204.107@tcp:12/10 lens 664/608 e 0 to 0 dl 1790901055 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5463.049804] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5464.862494] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5474.249529] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 20:30:57 (1790901057)
[ 5476.012306] LustreError: lustre-OST0000-osc-ffff8d5e842e3800: operation ost_write to node 192.168.204.107@tcp failed: rc = -107
[ 5476.027743] LustreError: Skipped 2 previous similar messages
[ 5476.046407] LustreError: lustre-OST0000-osc-ffff8d5e842e3800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5476.072328] Lustre: 2354:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.107@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5476.135417] LustreError: 73112:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff8d5e842e3800: namespace resource [0x240000400:0x3a62:0x0].0x0 (ffff8d5e9121eb00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5485.773974] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 20:31:08 (1790901068)
[ 5488.336374] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5490.453514] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 20:31:12 (1790901072)
[ 5492.410070] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5494.381770] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 20:31:17 (1790901077)
[ 5495.856287] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5497.490346] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 20:31:20 (1790901080)
[ 5499.033406] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5500.875672] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 20:31:23 (1790901083)
[ 5502.480608] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5504.014840] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 20:31:27 (1790901087)
[ 5505.606850] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5507.305415] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 20:31:30 (1790901090)
[ 5509.010994] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5510.911557] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 20:31:33 (1790901093)
[ 5512.379590] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5514.142403] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 20:31:36 (1790901096)
[ 5516.004140] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5518.801182] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 20:31:41 (1790901101)
[ 5520.026506] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5521.871783] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 20:31:44 (1790901104)
[ 5523.878932] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5525.558260] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 20:31:48 (1790901108)
[ 5527.955687] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5529.705476] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 20:31:52 (1790901112)
[ 5562.905186] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 20:32:25 (1790901145)
[ 5593.060489] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 20:32:55 (1790901175)
[ 5593.808150] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5604.452318] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 20:33:07 (1790901187)
[ 5605.252917] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5616.183219] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 20:33:19 (1790901199)
[ 5616.471318] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5625.251471] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 20:33:28 (1790901208)
[ 5625.608373] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5635.873839] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 20:33:38 (1790901218)
[ 5636.285293] Lustre: *** cfs_fail_loc=510, val=3***
[ 5644.684317] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 20:33:47 (1790901227)
[ 5644.967834] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5655.477082] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 20:33:58 (1790901238)
[ 5656.096628] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5725.232971] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 20:35:07 (1790901307)
[ 5725.822108] LustreError: 83318:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5728.592383] LustreError: lustre-MDT0000-mdc-ffff8d5e842e3800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5728.618229] LustreError: 83332:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8d5e842e3800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8d5e87439500) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5728.640667] LustreError: 83337:0:(file.c:6224:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5728.658126] Lustre: lustre-MDT0000-mdc-ffff8d5e842e3800: Connection restored to 192.168.204.107@tcp (at 192.168.204.107@tcp)
[ 5728.673891] LustreError: 83337:0:(file.c:6224:ll_inode_revalidate_fini()) Skipped 14 previous similar messages
[ 5728.681533] Lustre: Skipped 9 previous similar messages
[ 5729.896069] LustreError: 83318:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5732.028405] LustreError: 83348:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5735.000826] LustreError: lustre-MDT0000-mdc-ffff8d5e842e3800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5735.018766] LustreError: 83363:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8d5e842e3800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8d5e87439e00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5735.071569] LustreError: 83363:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5736.119927] LustreError: 83348:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5743.481467] LustreError: lustre-MDT0000-mdc-ffff8d5e842e3800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5744.003562] LustreError: 83404:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5746.934093] LustreError: lustre-MDT0000-mdc-ffff8d5e842e3800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5747.062929] LustreError: 83424:0:(file.c:6224:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5748.095138] LustreError: 83404:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5748.106783] LustreError: 83404:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8d5e842e3800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5749.339383] LustreError: 83442:0:(file.c:6224:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5749.366393] LustreError: 83442:0:(file.c:6224:ll_inode_revalidate_fini()) Skipped 12 previous similar messages
[ 5754.399248] LustreError: 83480:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5757.432131] LustreError: lustre-MDT0000-mdc-ffff8d5e842e3800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5757.463993] LustreError: 83494:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8d5e842e3800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8d5e8697a100) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5757.495912] LustreError: 83494:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5758.471510] LustreError: 83480:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5765.587928] LustreError: lustre-MDT0000-mdc-ffff8d5e842e3800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5766.137316] LustreError: 83536:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5768.935295] LustreError: lustre-MDT0000-mdc-ffff8d5e842e3800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5768.960649] LustreError: 83550:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8d5e842e3800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8d5e8697a800) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5768.996061] LustreError: 83550:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5770.210301] LustreError: 83536:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5773.308520] LustreError: lustre-MDT0000-mdc-ffff8d5e842e3800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5773.443846] LustreError: 83583:0:(file.c:6224:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5773.454363] LustreError: 83583:0:(file.c:6224:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5774.464047] LustreError: 83564:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8d5e842e3800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5782.587863] LustreError: 83635:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5782.600877] LustreError: 83635:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5786.048774] LustreError: lustre-MDT0000-mdc-ffff8d5e842e3800: operation ldlm_enqueue to node 192.168.204.107@tcp failed: rc = -107
[ 5786.056204] LustreError: Skipped 8 previous similar messages
[ 5786.067295] LustreError: lustre-MDT0000-mdc-ffff8d5e842e3800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5786.134839] LustreError: 83658:0:(file.c:6224:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5786.143751] LustreError: 83658:0:(file.c:6224:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5786.672631] LustreError: 83635:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5786.688482] LustreError: 83635:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5786.700270] LustreError: 83635:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8d5e842e3800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5797.760926] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 20:36:20 (1790901380)
[ 5816.470198] LustreError: 68568:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5816.487692] LustreError: 68568:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5820.559141] LustreError: 68568:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5820.571605] LustreError: 68568:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5828.159823] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 20:36:50 (1790901410)
[ 5880.626585] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 20:37:42 (1790901462)
[ 5943.603331] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 20:38:46 (1790901526)
[ 5978.944872] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 20:39:21 (1790901561)
[ 5979.505689] Lustre: Mounted lustre-client - version 2.17.58_63_g3705aed
[ 6104.774225] Lustre: Unmounted lustre-client
[ 6116.764281] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 20:41:38 (1790901698)
[ 6118.163666] LustreError: 87964:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 6122.065866] Lustre: lustre-OST0000-osc-ffff8d5e842e3800: Connection to lustre-OST0000 (at 192.168.204.107@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6122.083432] Lustre: Skipped 16 previous similar messages
[ 6122.108491] LustreError: lustre-OST0000-osc-ffff8d5e842e3800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6122.147781] LustreError: 88047:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff8d5e842e3800: namespace resource [0x240000400:0x3a8d:0x0].0x0 (ffff8d5e8697a200) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6122.188640] LustreError: 88047:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 6122.624925] LustreError: 87964:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout interrupted
[ 6122.640091] Lustre: 2352:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.107@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 6131.890390] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 20:41:53 (1790901713)
[ 6190.047407] Lustre: 88632:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790901719/real 1790901719]  req@ffff8d5e88287480 x1877890216286848/t0(0) o101->lustre-MDT0000-mdc-ffff8d5e842e3800@192.168.204.107@tcp:12/10 lens 328/344 e 0 to 1 dl 1790901774 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 6190.077282] Lustre: 88632:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 16 previous similar messages
[ 6199.733551] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 20:43:02 (1790901782)
[ 6202.356963] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 6204.630154] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 20:43:07 (1790901787)
[ 6271.737981] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6274.399252] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 20:44:16 (1790901856)
[ 6331.400652] Lustre: lustre-MDT0000-mdc-ffff8d5e842e3800: Connection restored to 192.168.204.107@tcp (at 192.168.204.107@tcp)
[ 6331.418844] Lustre: Skipped 13 previous similar messages
[ 6333.523835] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 20:45:15 (1790901915)
[ 6336.849440] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6339.701926] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 20:45:21 (1790901921)
[ 6341.863124] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6344.177805] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 20:45:26 (1790901926)
[ 6382.936973] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 20:46:05 (1790901965)
[ 6396.205124] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6424.543597] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 192.168.204.107@tcp) was lost; in progress operations using this service will fail
[ 6424.564452] LustreError: Skipped 2 previous similar messages
[ 6433.863959] Lustre: Evicted from MGS (at 192.168.204.107@tcp) after server handle changed from 0x5676da2a5e9c6a70 to 0x5676da2a5e9c825d
[ 6433.877473] Lustre: Skipped 2 previous similar messages
[ 6453.077972] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6454.949049] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6467.240259] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 20:47:29 (1790902049)
[ 6469.768403] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6471.946615] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 20:47:34 (1790902054)
[ 6487.016812] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 192.168.204.107@tcp) was lost; in progress operations using this service will fail
[ 6487.060050] Lustre: Evicted from MGS (at 192.168.204.107@tcp) after server handle changed from 0x5676da2a5e9c825d to 0x5676da2a5e9c8711
[ 6499.079962] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 20:48:01 (1790902081)
[ 6551.692957] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 20:48:54 (1790902134)
[ 6559.527183] LustreError: lustre-OST0000-osc-ffff8d5e842e3800: operation ost_setattr to node 192.168.204.107@tcp failed: rc = -107
[ 6559.556217] LustreError: Skipped 1 previous similar message
[ 6618.444625] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6621.998394] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6707.168599] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 192.168.204.107@tcp) was lost; in progress operations using this service will fail
[ 6707.193065] LustreError: Skipped 1 previous similar message
[ 6717.423789] Lustre: Evicted from MGS (at 192.168.204.107@tcp) after server handle changed from 0x5676da2a5e9c891e to 0x5676da2a5e9c9787
[ 6717.444283] Lustre: Skipped 1 previous similar message
[ 6731.621579] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6733.360828] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6759.405062] Lustre: lustre-MDT0000-mdc-ffff8d5e842e3800: Connection to lustre-MDT0000 (at 192.168.204.107@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6759.434225] Lustre: Skipped 8 previous similar messages
[ 6773.528341] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6775.220640] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6798.304503] Lustre: 2351:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790902322/real 1790902322]  req@ffff8d5e90867480 x1877890222832256/t0(0) o400->lustre-MDT0000-mdc-ffff8d5e842e3800@192.168.204.107@tcp:12/10 lens 224/224 e 0 to 1 dl 1790902382 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6798.350033] Lustre: 2351:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 25 previous similar messages
[ 6913.679249] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 20:54:56 (1790902496)
[ 6985.632444] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6991.734974] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 7245.895790] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 21:00:28 (1790902828)
[ 7384.054340] Lustre: lustre-MDT0000-mdc-ffff8d5e842e3800: Connection to lustre-MDT0000 (at 192.168.204.107@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7384.075889] Lustre: Skipped 1 previous similar message
[ 7394.289275] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 192.168.204.107@tcp) was lost; in progress operations using this service will fail
[ 7394.302869] LustreError: Skipped 1 previous similar message
[ 7394.404984] Lustre: Evicted from MGS (at 192.168.204.107@tcp) after server handle changed from 0x5676da2a5e9c9ecd to 0x5676da2a5eac3f77
[ 7394.412212] Lustre: Skipped 1 previous similar message
[ 7394.424877] Lustre: MGC192.168.204.107@tcp: Connection restored to 192.168.204.107@tcp (at 192.168.204.107@tcp)
[ 7394.431690] Lustre: Skipped 10 previous similar messages
[ 7425.992578] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 21:03:28 (1790903008)
[ 7427.241956] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7429.001665] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 21:03:31 (1790903011)
[ 7430.948075] LustreError: lustre-MDT0000-mdc-ffff8d5e842e3800: operation ldlm_enqueue to node 192.168.204.107@tcp failed: rc = -107
[ 7430.967520] LustreError: Skipped 281 previous similar messages
[ 7430.994791] LustreError: lustre-MDT0000-mdc-ffff8d5e842e3800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7438.349551] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 21:03:41 (1790903021)
[ 7604.915765] LustreError: lustre-OST0000-osc-ffff8d5e842e3800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7606.657649] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 21:06:29 (1790903189)
[ 7631.839539] Lustre: 2351:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790903196/real 1790903196]  req@ffff8d5eb0a20e00 x1877890234028928/t0(0) o4->lustre-OST0000-osc-ffff8d5e842e3800@192.168.204.107@tcp:6/4 lens 4584/448 e 0 to 1 dl 1790903216 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7631.882013] Lustre: 2351:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
[ 7650.491626] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 21:07:13 (1790903233)
[ 7652.033495] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7653.881404] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 21:07:16 (1790903236)
[ 7655.259724] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7656.995602] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 21:07:19 (1790903239)
[ 7694.050958] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 21:07:56 (1790903276)
[ 7733.228768] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 192.168.204.107@tcp) was lost; in progress operations using this service will fail
[ 7733.262827] Lustre: Evicted from MGS (at 192.168.204.107@tcp) after server handle changed from 0x5676da2a5eac3f77 to 0x5676da2a5eac660c
[ 7733.287674] Lustre: Skipped 1 previous similar message
[ 7749.301244] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 21:08:52 (1790903332)
[ 7751.274420] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7752.946236] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 21:08:55 (1790903335)
[ 7754.650197] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7756.236594] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 21:08:59 (1790903339)
[ 7766.038611] Lustre: Unmounted lustre-client
[ 7767.958402] Lustre: Mounted lustre-client - version 2.17.58_63_g3705aed
[ 7770.683138] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7812.476629] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2