[ 2054.440470] LustreError: 34833:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 2054.646439] LustreError: lustre-OST0000-osc-ffff8f9c42b78800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2064.456153] LustreError: 34833:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 awake
[ 2064.463714] Lustre: 2369:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.116@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 2064.472900] LustreError: 34833:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 2074.489277] LustreError: 34833:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 awake
[ 2074.494593] Lustre: 2367:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.116@tcp:/lustre/fid: [0x200000404:0x36:0x0]// may get corrupted (rc -108)
[ 2083.656718] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 01:04:25 (1790658265)
[ 2086.367386] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 2088.746802] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 01:04:30 (1790658270)
[ 2090.890146] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 2092.724560] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 01:04:34 (1790658274)
[ 2097.127795] LustreError: lustre-MDT0000-mdc-ffff8f9c42b78800: operation mds_close to node 192.168.202.116@tcp failed: rc = -19
[ 2117.153682] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 192.168.202.116@tcp) was lost; in progress operations using this service will fail
[ 2127.348883] Lustre: Evicted from MGS (at 192.168.202.116@tcp) after server handle changed from 0x4013bc47dda6ad0b to 0x4013bc47dda6c89b
[ 2220.603935] LustreError: lustre-MDT0000-mdc-ffff8f9c42b78800: operation ldlm_enqueue to node 192.168.202.116@tcp failed: rc = -19
[ 2220.615697] LustreError: Skipped 3 previous similar messages
[ 2238.945684] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 192.168.202.116@tcp) was lost; in progress operations using this service will fail
[ 2249.196396] Lustre: Evicted from MGS (at 192.168.202.116@tcp) after server handle changed from 0x4013bc47dda6c89b to 0x4013bc47dda87213
[ 2265.512474] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 01:07:27 (1790658447)
[ 2266.061132] Lustre: *** cfs_fail_loc=305, val=0***
[ 2266.065245] Lustre: Skipped 4 previous similar messages
[ 2309.600519] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 192.168.202.116@tcp) was lost; in progress operations using this service will fail
[ 2309.651212] Lustre: Evicted from MGS (at 192.168.202.116@tcp) after server handle changed from 0x4013bc47dda87213 to 0x4013bc47dda8774c
[ 2311.081730] Lustre: 15974:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.116@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2334.536380] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2336.882386] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2349.953943] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 01:08:51 (1790658531)
[ 2365.946967] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 192.168.202.116@tcp) was lost; in progress operations using this service will fail
[ 2365.991313] Lustre: Evicted from MGS (at 192.168.202.116@tcp) after server handle changed from 0x4013bc47dda8774c to 0x4013bc47dda87d50
[ 2366.062621] LustreError: lustre-MDT0000-mdc-ffff8f9c42b78800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2392.349479] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 01:09:34 (1790658574)
[ 2407.205372] LustreError: lustre-OST0000-osc-ffff8f9c42b78800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2429.608809] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 01:10:11 (1790658611)
[ 2443.808427] LustreError: lustre-MDT0000-mdc-ffff8f9c42b78800: operation mds_reint to node 192.168.202.116@tcp failed: rc = -19
[ 2462.176174] Lustre: 2367:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790658629/real 1790658629]  req@ffff8f9c434c7b80 x1877639153343104/t0(0) o400->MGC192.168.202.116@tcp@192.168.202.116@tcp:26/25 lens 224/224 e 0 to 1 dl 1790658645 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2462.206815] Lustre: 2367:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 17 previous similar messages
[ 2462.225013] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 192.168.202.116@tcp) was lost; in progress operations using this service will fail
[ 2472.431500] Lustre: Evicted from MGS (at 192.168.202.116@tcp) after server handle changed from 0x4013bc47dda87d50 to 0x4013bc47dda8c112
[ 2493.861670] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2496.844314] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2563.689310] Lustre: lustre-MDT0000-mdc-ffff8f9c42b78800: Connection to lustre-MDT0000 (at 192.168.202.116@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2563.701064] Lustre: Skipped 10 previous similar messages
[ 2584.032653] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 192.168.202.116@tcp) was lost; in progress operations using this service will fail
[ 2584.069136] Lustre: Evicted from MGS (at 192.168.202.116@tcp) after server handle changed from 0x4013bc47dda8c112 to 0x4013bc47ddaa5dfd
[ 2584.083426] Lustre: MGC192.168.202.116@tcp: Connection restored to 192.168.202.116@tcp (at 192.168.202.116@tcp)
[ 2584.090269] Lustre: Skipped 14 previous similar messages
[ 2585.223672] Lustre: 15974:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.116@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2613.059705] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2615.399318] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2700.768364] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 192.168.202.116@tcp) was lost; in progress operations using this service will fail
[ 2710.952885] Lustre: Evicted from MGS (at 192.168.202.116@tcp) after server handle changed from 0x4013bc47ddaa5dfd to 0x4013bc47ddabe2a0
[ 2727.617298] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2730.282381] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2759.701281] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 01:15:41 (1790658941)
[ 2764.586342] LustreError: lustre-MDT0000-mdc-ffff8f9c42b78800: operation ldlm_enqueue to node 192.168.202.116@tcp failed: rc = -19
[ 2764.596609] LustreError: Skipped 5 previous similar messages
[ 2782.178043] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 192.168.202.116@tcp) was lost; in progress operations using this service will fail
[ 2792.379065] Lustre: Evicted from MGS (at 192.168.202.116@tcp) after server handle changed from 0x4013bc47ddabe2a0 to 0x4013bc47ddacb4e6
[ 2795.083798] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2835.194843] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2872.430994] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2912.787504] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2955.296509] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 192.168.202.116@tcp) was lost; in progress operations using this service will fail
[ 2955.310308] LustreError: Skipped 3 previous similar messages
[ 2964.298066] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2964.537440] Lustre: Evicted from MGS (at 192.168.202.116@tcp) after server handle changed from 0x4013bc47ddad040f to 0x4013bc47ddad55df
[ 2964.553793] Lustre: Skipped 3 previous similar messages
[ 3033.865802] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 3086.816632] Lustre: 2370:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790659254/real 1790659254]  req@ffff8f9c58c83100 x1877639158246272/t0(0) o400->MGC192.168.202.116@tcp@192.168.202.116@tcp:26/25 lens 224/224 e 0 to 1 dl 1790659270 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3086.855716] Lustre: 2370:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
[ 3131.182597] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 01:21:53 (1790659313)
[ 3190.252060] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3192.542203] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3478.807439] LustreError: lustre-OST0000-osc-ffff8f9c42b78800: operation ldlm_enqueue to node 192.168.202.116@tcp failed: rc = -19
[ 3478.828268] LustreError: Skipped 5 previous similar messages
[ 3478.840439] Lustre: lustre-OST0000-osc-ffff8f9c42b78800: Connection to lustre-OST0000 (at 192.168.202.116@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3478.850078] Lustre: Skipped 7 previous similar messages
[ 3520.461717] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3522.986818] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3846.947423] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3848.899683] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4104.979572] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 01:38:07 (1790660287)
[ 4123.104519] Lustre: 50648:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790660290/real 1790660290]  req@ffff8f9c44951180 x1877639175937408/t0(0) o101->lustre-MDT0000-mdc-ffff8f9c42b78800@192.168.202.116@tcp:12/10 lens 576/1152 e 0 to 1 dl 1790660306 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 4123.152139] Lustre: lustre-MDT0000-mdc-ffff8f9c42b78800: Connection to lustre-MDT0000 (at 192.168.202.116@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4123.167693] Lustre: Skipped 1 previous similar message
[ 4123.198119] Lustre: lustre-MDT0000-mdc-ffff8f9c42b78800: Connection restored to 192.168.202.116@tcp (at 192.168.202.116@tcp)
[ 4123.206660] Lustre: Skipped 16 previous similar messages
[ 4130.775080] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 01:38:33 (1790660313)
[ 4158.696961] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 01:39:00 (1790660340)
[ 4186.699941] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 01:39:28 (1790660368)
[ 4187.275284] Lustre: Mounted lustre-client - version 2.17.58_63_g5548f20
[ 4219.373852] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 192.168.202.116@tcp) was lost; in progress operations using this service will fail
[ 4219.389509] LustreError: Skipped 2 previous similar messages
[ 4219.414394] Lustre: Evicted from MGS (at 192.168.202.116@tcp) after server handle changed from 0x4013bc47ddae8214 to 0x4013bc47ddc34066
[ 4219.425424] Lustre: Skipped 2 previous similar messages
[ 4219.429842] Lustre: MGC192.168.202.116@tcp: Connection restored to 192.168.202.116@tcp (at 192.168.202.116@tcp)
[ 4219.437658] Lustre: Skipped 2 previous similar messages
[ 4233.690684] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4235.142475] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4237.356504] Lustre: Unmounted lustre-client
[ 4244.519072] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 01:40:26 (1790660426)
[ 4382.507707] Lustre: lustre-OST0000-osc-ffff8f9c42b78800: Connection restored to 192.168.202.116@tcp (at 192.168.202.116@tcp)
[ 4382.533078] Lustre: Skipped 8 previous similar messages
[ 4539.975345] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 01:45:22 (1790660722)
[ 4590.164383] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 01:46:12 (1790660772)
[ 4593.984801] Lustre: Unmounted lustre-client
[ 4616.530293] Lustre: Mounted lustre-client - version 2.17.58_63_g5548f20
[ 4623.761441] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 01:46:46 (1790660806)
[ 4624.034284] LustreError: 56572:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4625.080132] LustreError: 56572:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4625.271268] Lustre: *** cfs_fail_loc=305, val=0***
[ 4648.465208] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 01:47:10 (1790660830)
[ 4649.091073] Lustre: Mounted lustre-client - version 2.17.58_63_g5548f20
[ 4652.114401] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4662.455255] Lustre: Unmounted lustre-client
[ 4672.290360] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 01:47:33 (1790660853)
[ 4759.686345] LustreError: lustre-MDT0000-mdc-ffff8f9c509db000: operation mds_reint to node 192.168.202.116@tcp failed: rc = -19
[ 4759.693399] LustreError: Skipped 2 previous similar messages
[ 4759.696213] Lustre: lustre-MDT0000-mdc-ffff8f9c509db000: Connection to lustre-MDT0000 (at 192.168.202.116@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4759.714548] Lustre: Skipped 23 previous similar messages
[ 4778.467724] Lustre: 2369:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790660945/real 1790660945]  req@ffff8f9c47c2d880 x1877639178662912/t0(0) o400->MGC192.168.202.116@tcp@192.168.202.116@tcp:26/25 lens 224/224 e 0 to 1 dl 1790660961 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4778.495940] Lustre: 2369:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 136 previous similar messages
[ 4778.511100] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 192.168.202.116@tcp) was lost; in progress operations using this service will fail
[ 4778.538820] Lustre: Evicted from MGS (at 192.168.202.116@tcp) after server handle changed from 0x4013bc47ddc35910 to 0x4013bc47ddc60c18
[ 4778.551793] Lustre: MGC192.168.202.116@tcp: Connection restored to 192.168.202.116@tcp (at 192.168.202.116@tcp)
[ 4778.560995] Lustre: Skipped 11 previous similar messages
[ 4930.816613] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 01:51:53 (1790661113)
[ 4936.622582] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4955.618070] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 192.168.202.116@tcp) was lost; in progress operations using this service will fail
[ 4955.653540] Lustre: Evicted from MGS (at 192.168.202.116@tcp) after server handle changed from 0x4013bc47ddc60c18 to 0x4013bc47ddcad8bb
[ 4955.701570] LustreError: lustre-MDT0000-mdc-ffff8f9c509db000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4973.689920] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 01:52:35 (1790661155)
[ 4974.250729] Lustre: Mounted lustre-client - version 2.17.58_63_g5548f20
[ 4981.732454] LustreError: lustre-OST0000-osc-ffff8f9c509db000: operation ldlm_enqueue to node 192.168.202.116@tcp failed: rc = -107
[ 4981.763172] LustreError: lustre-OST0000-osc-ffff8f9c509db000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4995.062180] Lustre: Unmounted lustre-client
[ 5002.984314] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 01:53:04 (1790661184)
[ 5010.178778] LustreError: lustre-MDT0000-mdc-ffff8f9c509db000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5010.210863] LustreError: 60965:0:(file.c:6224:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 5018.599642] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 01:53:20 (1790661200)
[ 5018.864623] LustreError: 61669:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 5024.224262] LustreError: 61669:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=0
[ 5024.229761] LustreError: 61669:0:(import.c:756:ptlrpc_connect_import_locked()) already connected
[ 5024.964408] LustreError: lustre-MDT0000-mdc-ffff8f9c509db000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5024.983452] LustreError: 61686:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 5025.050837] LustreError: 61691:0:(file.c:6224:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5026.174303] LustreError: 61698:0:(file.c:6224:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5026.183854] LustreError: 61698:0:(file.c:6224:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5028.487615] LustreError: 61720:0:(file.c:6224:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5028.508902] LustreError: 61720:0:(file.c:6224:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5033.182338] LustreError: 61765:0:(file.c:6224:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5033.195511] LustreError: 61765:0:(file.c:6224:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 5043.478308] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 01:53:45 (1790661225)
[ 5083.146525] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5084.996325] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5095.618278] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 01:54:37 (1790661277)
[ 5135.311160] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5136.542830] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5147.387831] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 01:55:29 (1790661329)
[ 5215.822294] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5217.684291] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5227.955764] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 01:56:49 (1790661409)
[ 5270.357389] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5271.837447] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5277.075493] Lustre: Unmounted lustre-client
[ 5333.398280] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5335.935331] Lustre: Mounted lustre-client - version 2.17.58_63_g5548f20
[ 5342.377741] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 01:58:44 (1790661524)
[ 5345.371253] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5346.966947] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 01:58:49 (1790661529)
[ 5378.482754] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 01:59:20 (1790661560)
[ 5380.619327] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5382.642994] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 01:59:24 (1790661564)
[ 5384.357645] Lustre: *** cfs_fail_loc=805, val=0***
[ 5384.414837] Lustre: Mounted lustre-client - version 2.17.58_63_g5548f20
[ 5390.410764] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5411.301503] Lustre: 2368:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790661578/real 1790661578]  req@ffff8f9c43e76d80 x1877639180716672/t0(0) o400->MGC192.168.202.116@tcp@192.168.202.116@tcp:26/25 lens 224/224 e 0 to 1 dl 1790661594 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5411.344232] Lustre: 2368:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 5411.355243] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 192.168.202.116@tcp) was lost; in progress operations using this service will fail
[ 5411.378721] Lustre: lustre-MDT0000-mdc-ffff8f9c43284000: Connection to lustre-MDT0000 (at 192.168.202.116@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5411.398412] Lustre: Skipped 13 previous similar messages
[ 5420.562142] Lustre: Evicted from MGS (at 192.168.202.116@tcp) after server handle changed from 0x4013bc47ddcae2e0 to 0x4013bc47ddcae510
[ 5420.592547] Lustre: MGC192.168.202.116@tcp: Connection restored to 192.168.202.116@tcp (at 192.168.202.116@tcp)
[ 5420.615467] Lustre: Skipped 13 previous similar messages
[ 5433.389531] Lustre: Unmounted lustre-client
[ 5443.227191] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 02:00:24 (1790661624)
[ 5486.051333] LustreError: 2366:0:(client.c:3449:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff8f9c47a88000 x1877639180729088/t85899345925(85899345925) o101->lustre-MDT0000-mdc-ffff8f9c44ae2000@192.168.202.116@tcp:12/10 lens 648/608 e 0 to 0 dl 1790661685 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5496.668371] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5498.347751] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5508.131564] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 02:01:29 (1790661689)
[ 5510.111905] LustreError: lustre-OST0000-osc-ffff8f9c44ae2000: operation ost_write to node 192.168.202.116@tcp failed: rc = -107
[ 5510.122856] LustreError: Skipped 2 previous similar messages
[ 5510.143354] LustreError: lustre-OST0000-osc-ffff8f9c44ae2000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5510.153837] Lustre: 2370:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.116@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5510.207301] LustreError: 72883:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff8f9c44ae2000: namespace resource [0x240000400:0x3c82:0x0].0x0 (ffff8f9c47cb1800) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5519.624495] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 02:01:41 (1790661701)
[ 5521.310296] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5523.313157] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 02:01:45 (1790661705)
[ 5525.015947] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5526.533369] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 02:01:48 (1790661708)
[ 5528.328424] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5531.330787] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 02:01:52 (1790661712)
[ 5533.484865] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5536.085498] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 02:01:57 (1790661717)
[ 5537.947159] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5539.830469] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 02:02:01 (1790661721)
[ 5541.664370] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5544.193503] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 02:02:05 (1790661725)
[ 5545.972173] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5548.178728] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 02:02:09 (1790661729)
[ 5550.508643] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5552.473201] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 02:02:14 (1790661734)
[ 5554.302667] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5556.565264] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 02:02:18 (1790661738)
[ 5558.688816] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5560.210697] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 02:02:22 (1790661742)
[ 5562.259968] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5564.812855] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 02:02:26 (1790661746)
[ 5567.917411] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5569.970193] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 02:02:31 (1790661751)
[ 5608.253276] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 02:03:10 (1790661790)
[ 5638.258396] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 02:03:40 (1790661820)
[ 5638.786241] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5649.482817] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 02:03:51 (1790661831)
[ 5649.773271] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5660.367522] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 02:04:02 (1790661842)
[ 5660.699665] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5670.541101] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 02:04:12 (1790661852)
[ 5671.017794] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5680.805427] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 02:04:22 (1790661862)
[ 5681.181330] Lustre: *** cfs_fail_loc=510, val=3***
[ 5691.728374] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 02:04:33 (1790661873)
[ 5692.237341] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5704.124419] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 02:04:46 (1790661886)
[ 5704.585460] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5772.466241] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 02:05:54 (1790661954)
[ 5772.947206] LustreError: 83095:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5776.025262] LustreError: lustre-MDT0000-mdc-ffff8f9c44ae2000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5776.042742] LustreError: 83110:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8f9c44ae2000: inode [0x20000a042:0xb:0x0] mdc close failed: rc = -108
[ 5776.081123] LustreError: 83110:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8f9c44ae2000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8f9c52863700) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5776.953507] LustreError: 83095:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5779.121236] LustreError: 83128:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5782.258295] LustreError: lustre-MDT0000-mdc-ffff8f9c44ae2000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5782.284720] LustreError: 83142:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8f9c44ae2000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8f9c52863200) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5782.328662] LustreError: 83142:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5783.201491] LustreError: 83128:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5790.292778] LustreError: lustre-MDT0000-mdc-ffff8f9c44ae2000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5790.805247] LustreError: 83183:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5793.937964] LustreError: lustre-MDT0000-mdc-ffff8f9c44ae2000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5794.031040] LustreError: 83203:0:(file.c:6224:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5794.044396] LustreError: 83203:0:(file.c:6224:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5794.888299] LustreError: 83183:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5794.897171] LustreError: 83183:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8f9c44ae2000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5795.229712] LustreError: 83209:0:(file.c:6224:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5795.241168] LustreError: 83209:0:(file.c:6224:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5797.566256] LustreError: 83232:0:(file.c:6224:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5797.592363] LustreError: 83232:0:(file.c:6224:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5801.601588] LustreError: 83259:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5804.719809] LustreError: lustre-MDT0000-mdc-ffff8f9c44ae2000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5804.739762] LustreError: 83273:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8f9c44ae2000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8f9c5048a800) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5804.785533] LustreError: 83273:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5805.689271] LustreError: 83259:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5812.828169] LustreError: lustre-MDT0000-mdc-ffff8f9c44ae2000: operation ldlm_enqueue to node 192.168.202.116@tcp failed: rc = -107
[ 5812.844860] LustreError: Skipped 5 previous similar messages
[ 5812.855534] LustreError: lustre-MDT0000-mdc-ffff8f9c44ae2000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5813.341879] LustreError: 83315:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5816.525465] LustreError: lustre-MDT0000-mdc-ffff8f9c44ae2000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5816.550539] LustreError: 83329:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8f9c44ae2000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8f9c5048a400) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5816.568328] LustreError: 83329:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5817.436525] LustreError: 83315:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5820.671318] LustreError: lustre-MDT0000-mdc-ffff8f9c44ae2000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5820.761933] LustreError: 83362:0:(file.c:6224:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5820.770087] LustreError: 83362:0:(file.c:6224:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5821.633300] LustreError: 83343:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8f9c44ae2000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5830.002742] LustreError: 83414:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5830.015149] LustreError: 83414:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5834.020019] LustreError: 83414:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5834.026863] LustreError: 83414:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5834.422690] LustreError: lustre-MDT0000-mdc-ffff8f9c44ae2000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5841.622820] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 02:07:03 (1790662023)
[ 5861.989186] LustreError: 68330:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5866.064181] LustreError: 68330:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5873.715171] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 02:07:35 (1790662055)
[ 5926.676271] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 02:08:28 (1790662108)
[ 5992.116576] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 02:09:34 (1790662174)
[ 6027.298764] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 02:10:09 (1790662209)
[ 6028.215482] Lustre: Mounted lustre-client - version 2.17.58_63_g5548f20
[ 6152.060178] Lustre: Unmounted lustre-client
[ 6160.218892] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 02:12:22 (1790662342)
[ 6161.270754] LustreError: 87698:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 6163.948907] Lustre: lustre-OST0000-osc-ffff8f9c44ae2000: Connection to lustre-OST0000 (at 192.168.202.116@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6163.964292] Lustre: Skipped 17 previous similar messages
[ 6163.981103] LustreError: lustre-OST0000-osc-ffff8f9c44ae2000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6163.998712] LustreError: 87780:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff8f9c44ae2000: namespace resource [0x240000400:0x3cad:0x0].0x0 (ffff8f9c52863b00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6164.037488] LustreError: 87780:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 6164.352227] LustreError: 87698:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout interrupted
[ 6164.361328] Lustre: 2368:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.116@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 6170.862493] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 02:12:33 (1790662353)
[ 6228.960182] Lustre: 88364:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790662357/real 1790662357]  req@ffff8f9c507a1c00 x1877639180900480/t0(0) o101->lustre-MDT0000-mdc-ffff8f9c44ae2000@192.168.202.116@tcp:12/10 lens 328/344 e 0 to 1 dl 1790662412 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 6228.993063] Lustre: 88364:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 21 previous similar messages
[ 6229.028839] Lustre: lustre-MDT0000-mdc-ffff8f9c44ae2000: Connection restored to 192.168.202.116@tcp (at 192.168.202.116@tcp)
[ 6229.037450] Lustre: Skipped 18 previous similar messages
[ 6236.204225] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 02:13:38 (1790662418)
[ 6238.394112] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 6240.724101] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 02:13:42 (1790662422)
[ 6303.097130] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6304.384818] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 02:14:46 (1790662486)
[ 6362.382713] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 02:15:44 (1790662544)
[ 6365.377865] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6367.710547] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 02:15:49 (1790662549)
[ 6369.475840] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6371.172941] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 02:15:53 (1790662553)
[ 6409.252786] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 02:16:31 (1790662591)
[ 6422.206378] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6449.120395] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 192.168.202.116@tcp) was lost; in progress operations using this service will fail
[ 6449.147765] LustreError: Skipped 2 previous similar messages
[ 6459.374342] Lustre: Evicted from MGS (at 192.168.202.116@tcp) after server handle changed from 0x4013bc47ddcaf111 to 0x4013bc47ddcb08fe
[ 6459.390460] Lustre: Skipped 2 previous similar messages
[ 6461.000394] Lustre: 68332:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.116@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6477.603744] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6479.396185] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6489.390762] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 02:17:51 (1790662671)
[ 6491.309556] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6492.883661] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 02:17:55 (1790662675)
[ 6510.579671] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 192.168.202.116@tcp) was lost; in progress operations using this service will fail
[ 6510.621309] Lustre: Evicted from MGS (at 192.168.202.116@tcp) after server handle changed from 0x4013bc47ddcb08fe to 0x4013bc47ddcb0db9
[ 6519.239721] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 02:18:21 (1790662701)
[ 6556.679684] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 02:18:58 (1790662738)
[ 6562.648958] LustreError: lustre-OST0000-osc-ffff8f9c44ae2000: operation ost_setattr to node 192.168.202.116@tcp failed: rc = -19
[ 6562.656896] LustreError: Skipped 7 previous similar messages
[ 6614.095923] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6617.357369] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6704.126586] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 192.168.202.116@tcp) was lost; in progress operations using this service will fail
[ 6704.140274] LustreError: Skipped 1 previous similar message
[ 6704.164494] Lustre: Evicted from MGS (at 192.168.202.116@tcp) after server handle changed from 0x4013bc47ddcb10c2 to 0x4013bc47ddcb1e21
[ 6704.171101] Lustre: Skipped 1 previous similar message
[ 6717.212544] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6718.726357] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6746.556466] Lustre: 68332:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.116@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6764.807649] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6766.444513] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6890.145120] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 02:24:32 (1790663072)
[ 6902.197052] Lustre: lustre-OST0000-osc-ffff8f9c44ae2000: Connection to lustre-OST0000 (at 192.168.202.116@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6902.225220] Lustre: Skipped 9 previous similar messages
[ 6960.088557] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6964.318544] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 7234.545364] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 02:30:16 (1790663416)
[ 7392.234980] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 192.168.202.116@tcp) was lost; in progress operations using this service will fail
[ 7392.247678] LustreError: Skipped 1 previous similar message
[ 7392.264515] Lustre: Evicted from MGS (at 192.168.202.116@tcp) after server handle changed from 0x4013bc47ddcb22ff to 0x4013bc47dddac6f8
[ 7392.269575] Lustre: Skipped 1 previous similar message
[ 7392.277086] Lustre: MGC192.168.202.116@tcp: Connection restored to 192.168.202.116@tcp (at 192.168.202.116@tcp)
[ 7392.287544] Lustre: Skipped 12 previous similar messages
[ 7428.746590] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 02:33:30 (1790663610)
[ 7430.022668] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7431.806253] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 02:33:33 (1790663613)
[ 7433.884849] LustreError: lustre-MDT0000-mdc-ffff8f9c44ae2000: operation ldlm_enqueue to node 192.168.202.116@tcp failed: rc = -107
[ 7433.894523] LustreError: Skipped 298 previous similar messages
[ 7433.903937] LustreError: lustre-MDT0000-mdc-ffff8f9c44ae2000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7441.000327] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 02:33:43 (1790663623)
[ 7608.317458] LustreError: lustre-OST0000-osc-ffff8f9c44ae2000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7610.082714] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 02:36:32 (1790663792)
[ 7635.936270] Lustre: 2370:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790663799/real 1790663799]  req@ffff8f9c4d942300 x1877639198640640/t0(0) o4->lustre-OST0000-osc-ffff8f9c44ae2000@192.168.202.116@tcp:6/4 lens 4584/448 e 0 to 1 dl 1790663819 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7635.973855] Lustre: 2370:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 26 previous similar messages
[ 7635.991441] Lustre: lustre-OST0000-osc-ffff8f9c44ae2000: Connection to lustre-OST0000 (at 192.168.202.116@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7636.010833] Lustre: Skipped 3 previous similar messages
[ 7654.394651] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 02:37:16 (1790663836)
[ 7655.758643] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7657.712630] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 02:37:19 (1790663839)
[ 7659.074262] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7660.468434] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 02:37:22 (1790663842)
[ 7695.492721] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 02:37:57 (1790663877)
[ 7733.231165] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 192.168.202.116@tcp) was lost; in progress operations using this service will fail
[ 7733.285201] Lustre: Evicted from MGS (at 192.168.202.116@tcp) after server handle changed from 0x4013bc47dddac6f8 to 0x4013bc47dddaeda9
[ 7733.310045] Lustre: Skipped 1 previous similar message
[ 7749.073397] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 02:38:50 (1790663930)
[ 7751.288489] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7753.634498] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 02:38:55 (1790663935)
[ 7755.140465] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7756.617849] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 02:38:58 (1790663938)
[ 7762.936091] Lustre: 2369:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790663906/real 1790663906]  req@ffff8f9c47230700 x1877639198796416/t0(0) o400->lustre-MDT0000-mdc-ffff8f9c44ae2000@192.168.202.116@tcp:12/10 lens 224/224 e 0 to 1 dl 1790663946 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7769.366243] Lustre: Unmounted lustre-client
[ 7771.309449] Lustre: Mounted lustre-client - version 2.17.58_63_g5548f20
[ 7774.301861] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7805.454548] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2