[ 1808.588424] LustreError: lustre-OST0000-osc-ffff9a1e12111000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1808.601728] Lustre: 2355:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.151@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1813.832433] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 05:19:47 (1789636787)
[ 1815.505555] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1816.874655] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 05:19:50 (1789636790)
[ 1818.153733] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1819.597364] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 05:19:53 (1789636793)
[ 1822.281382] LustreError: lustre-MDT0000-mdc-ffff9a1e12111000: operation ldlm_enqueue to node 192.168.204.151@tcp failed: rc = -19
[ 1822.291231] LustreError: Skipped 2 previous similar messages
[ 1839.602659] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 192.168.204.151@tcp) was lost; in progress operations using this service will fail
[ 1839.631935] Lustre: Evicted from MGS (at 192.168.204.151@tcp) after server handle changed from 0x22220beb027a59c5 to 0x22220beb027a782d
[ 1938.533255] LustreError: lustre-MDT0000-mdc-ffff9a1e12111000: operation ldlm_enqueue to node 192.168.204.151@tcp failed: rc = -19
[ 1957.354873] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 192.168.204.151@tcp) was lost; in progress operations using this service will fail
[ 1957.398052] Lustre: Evicted from MGS (at 192.168.204.151@tcp) after server handle changed from 0x22220beb027a782d to 0x22220beb027dc321
[ 1964.770773] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 05:22:18 (1789636938)
[ 1965.090111] Lustre: *** cfs_fail_loc=305, val=0***
[ 1965.092928] Lustre: Skipped 2 previous similar messages
[ 2003.430258] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 192.168.204.151@tcp) was lost; in progress operations using this service will fail
[ 2003.451720] Lustre: Evicted from MGS (at 192.168.204.151@tcp) after server handle changed from 0x22220beb027dc321 to 0x22220beb027de158
[ 2012.389810] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2013.390271] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2019.541937] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 05:23:12 (1789636992)
[ 2029.036630] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 192.168.204.151@tcp) was lost; in progress operations using this service will fail
[ 2029.059167] Lustre: Evicted from MGS (at 192.168.204.151@tcp) after server handle changed from 0x22220beb027de158 to 0x22220beb027de52c
[ 2029.095726] LustreError: lustre-MDT0000-mdc-ffff9a1e12111000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2043.863600] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 05:23:37 (1789637017)
[ 2054.643214] LustreError: lustre-OST0000-osc-ffff9a1e12111000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2065.445785] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 05:23:59 (1789637039)
[ 2077.704583] LustreError: lustre-MDT0000-mdc-ffff9a1e12111000: operation mds_reint to node 192.168.204.151@tcp failed: rc = -19
[ 2077.716399] LustreError: Skipped 2 previous similar messages
[ 2094.564340] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 192.168.204.151@tcp) was lost; in progress operations using this service will fail
[ 2094.582760] Lustre: Evicted from MGS (at 192.168.204.151@tcp) after server handle changed from 0x22220beb027de52c to 0x22220beb027e55fe
[ 2105.536673] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2107.109502] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2170.858960] Lustre: lustre-MDT0000-mdc-ffff9a1e12111000: Connection to lustre-MDT0000 (at 192.168.204.151@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2170.874320] Lustre: Skipped 14 previous similar messages
[ 2186.724338] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 192.168.204.151@tcp) was lost; in progress operations using this service will fail
[ 2186.764442] Lustre: Evicted from MGS (at 192.168.204.151@tcp) after server handle changed from 0x22220beb027e55fe to 0x22220beb0280f0a2
[ 2186.781703] Lustre: MGC192.168.204.151@tcp: Connection restored to 192.168.204.151@tcp (at 192.168.204.151@tcp)
[ 2186.787152] Lustre: Skipped 19 previous similar messages
[ 2211.606249] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2213.482124] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2276.931189] LustreError: lustre-MDT0000-mdc-ffff9a1e12111000: operation mds_close to node 192.168.204.151@tcp failed: rc = -19
[ 2276.941632] LustreError: Skipped 2 previous similar messages
[ 2293.221665] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 192.168.204.151@tcp) was lost; in progress operations using this service will fail
[ 2293.255944] Lustre: Evicted from MGS (at 192.168.204.151@tcp) after server handle changed from 0x22220beb0280f0a2 to 0x22220beb02837781
[ 2301.440441] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2302.650619] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2328.991825] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 05:28:22 (1789637302)
[ 2352.702811] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2374.129300] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 192.168.204.151@tcp) was lost; in progress operations using this service will fail
[ 2374.138404] LustreError: Skipped 1 previous similar message
[ 2374.158455] Lustre: Evicted from MGS (at 192.168.204.151@tcp) after server handle changed from 0x22220beb0284c669 to 0x22220beb02850848
[ 2374.171184] Lustre: Skipped 1 previous similar message
[ 2376.614072] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2404.344377] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2434.528541] Lustre: 2355:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789637393/real 1789637393]  req@ffff9a1e1249c700 x1876568359840896/t0(0) o400->MGC192.168.204.151@tcp@192.168.204.151@tcp:26/25 lens 224/224 e 0 to 1 dl 1789637409 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2434.567156] Lustre: 2355:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 9 previous similar messages
[ 2442.040692] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2485.850926] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2528.754086] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 192.168.204.151@tcp) was lost; in progress operations using this service will fail
[ 2528.771376] LustreError: Skipped 3 previous similar messages
[ 2528.780835] Lustre: Evicted from MGS (at 192.168.204.151@tcp) after server handle changed from 0x22220beb02864f18 to 0x22220beb02875918
[ 2528.786148] Lustre: Skipped 3 previous similar messages
[ 2534.947219] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2567.122795] LustreError: lustre-MDT0000-mdc-ffff9a1e12111000: operation mds_reint to node 192.168.204.151@tcp failed: rc = -19
[ 2567.130054] LustreError: Skipped 8 previous similar messages
[ 2611.473201] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 05:33:05 (1789637585)
[ 2649.915611] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 2651.072624] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 2957.641942] Lustre: lustre-OST0000-osc-ffff9a1e12111000: Connection to lustre-OST0000 (at 192.168.204.151@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2957.674096] Lustre: Skipped 9 previous similar messages
[ 2979.452560] Lustre: lustre-OST0000-osc-ffff9a1e12111000: Connection restored to 192.168.204.151@tcp (at 192.168.204.151@tcp)
[ 2979.463675] Lustre: Skipped 18 previous similar messages
[ 2997.596090] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 2999.453915] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3291.486499] LustreError: lustre-OST0000-osc-ffff9a1e12111000: operation ldlm_enqueue to node 192.168.204.151@tcp failed: rc = -19
[ 3291.504770] LustreError: Skipped 3 previous similar messages
[ 3335.018709] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3337.861907] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3586.178535] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 05:49:19 (1789638559)
[ 3604.448431] Lustre: 50611:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789638562/real 1789638562]  req@ffff9a1e034c9880 x1876568379854976/t0(0) o101->lustre-MDT0000-mdc-ffff9a1e12111000@192.168.204.151@tcp:12/10 lens 576/1152 e 0 to 1 dl 1789638578 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3604.487243] Lustre: lustre-MDT0000-mdc-ffff9a1e12111000: Connection to lustre-MDT0000 (at 192.168.204.151@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3604.530363] Lustre: Skipped 1 previous similar message
[ 3604.618473] Lustre: lustre-MDT0000-mdc-ffff9a1e12111000: Connection restored to 192.168.204.151@tcp (at 192.168.204.151@tcp)
[ 3604.632387] Lustre: Skipped 1 previous similar message
[ 3616.910526] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 05:49:49 (1789638589)
[ 3644.655385] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 05:50:17 (1789638617)
[ 3671.247398] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 05:50:44 (1789638644)
[ 3671.813501] Lustre: Mounted lustre-client - version 2.17.58_39_ga8a99f2
[ 3702.560523] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 192.168.204.151@tcp) was lost; in progress operations using this service will fail
[ 3702.575847] LustreError: Skipped 1 previous similar message
[ 3713.022990] Lustre: Evicted from MGS (at 192.168.204.151@tcp) after server handle changed from 0x22220beb0288acde to 0x22220beb029d7ad4
[ 3713.032667] Lustre: Skipped 1 previous similar message
[ 3720.777796] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 3722.456631] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3724.705875] Lustre: Unmounted lustre-client
[ 3731.120940] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 05:51:44 (1789638704)
[ 4030.016560] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 05:56:42 (1789639002)
[ 4079.465072] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 05:57:32 (1789639052)
[ 4082.436657] Lustre: Unmounted lustre-client
[ 4104.838532] Lustre: Mounted lustre-client - version 2.17.58_39_ga8a99f2
[ 4110.742930] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 05:58:04 (1789639084)
[ 4111.039993] LustreError: 56540:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4112.097041] LustreError: 56540:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4112.257401] Lustre: *** cfs_fail_loc=305, val=0***
[ 4134.241288] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 05:58:27 (1789639107)
[ 4134.577544] Lustre: Mounted lustre-client - version 2.17.58_39_ga8a99f2
[ 4135.957135] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4146.263929] Lustre: Unmounted lustre-client
[ 4152.589512] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 05:58:46 (1789639126)
[ 4233.667747] LustreError: lustre-MDT0000-mdc-ffff9a1e03582800: operation mds_reint to node 192.168.204.151@tcp failed: rc = -19
[ 4233.675903] Lustre: lustre-MDT0000-mdc-ffff9a1e03582800: Connection to lustre-MDT0000 (at 192.168.204.151@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4233.686934] Lustre: Skipped 23 previous similar messages
[ 4252.653826] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 192.168.204.151@tcp) was lost; in progress operations using this service will fail
[ 4252.688491] Lustre: Evicted from MGS (at 192.168.204.151@tcp) after server handle changed from 0x22220beb029d96b1 to 0x22220beb02a05233
[ 4252.699885] Lustre: MGC192.168.204.151@tcp: Connection restored to 192.168.204.151@tcp (at 192.168.204.151@tcp)
[ 4252.712039] Lustre: Skipped 24 previous similar messages
[ 4406.106901] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 06:02:58 (1789639378)
[ 4411.746475] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4425.720652] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 192.168.204.151@tcp) was lost; in progress operations using this service will fail
[ 4425.753955] Lustre: Evicted from MGS (at 192.168.204.151@tcp) after server handle changed from 0x22220beb02a05233 to 0x22220beb02a511d2
[ 4436.042620] LustreError: lustre-MDT0000-mdc-ffff9a1e03582800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4444.016773] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 06:03:37 (1789639417)
[ 4444.497468] Lustre: Mounted lustre-client - version 2.17.58_39_ga8a99f2
[ 4451.936437] LustreError: lustre-OST0000-osc-ffff9a1e03582800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4462.048396] Lustre: 2355:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789639420/real 1789639420]  req@ffff9a1e06643100 x1876568384561536/t0(0) o103->lustre-OST0000-osc-ffff9a1e12069800@192.168.204.151@tcp:17/18 lens 328/224 e 0 to 1 dl 1789639436 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295
[ 4462.082402] Lustre: 2355:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 155 previous similar messages
[ 4464.434571] Lustre: Unmounted lustre-client
[ 4471.500061] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 06:04:04 (1789639444)
[ 4478.911363] LustreError: lustre-MDT0000-mdc-ffff9a1e03582800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4478.940740] LustreError: 60936:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4488.776117] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 06:04:21 (1789639461)
[ 4489.141832] LustreError: 61641:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4493.089898] LustreError: lustre-MDT0000-mdc-ffff9a1e03582800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4493.103441] LustreError: 61658:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4493.119350] LustreError: 61641:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=1023
[ 4493.137579] LustreError: 61641:0:(import.c:762:ptlrpc_connect_import_locked()) already connecting
[ 4493.203086] LustreError: 61663:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4494.380957] LustreError: 61669:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4494.396157] LustreError: 61669:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4496.921237] LustreError: 61692:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4496.948916] LustreError: 61692:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4501.596520] LustreError: 61737:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4501.606686] LustreError: 61737:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4511.259302] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 06:04:44 (1789639484)
[ 4554.137472] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4555.762955] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4566.564779] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 06:05:39 (1789639539)
[ 4610.925924] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4612.672608] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4623.213312] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 06:06:36 (1789639596)
[ 4686.073827] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4687.408845] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4694.695378] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 06:07:47 (1789639667)
[ 4736.494882] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4738.554358] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4743.870643] Lustre: Unmounted lustre-client
[ 4794.327498] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4796.703522] Lustre: Mounted lustre-client - version 2.17.58_39_ga8a99f2
[ 4803.716854] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 06:09:36 (1789639776)
[ 4807.610902] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 4809.240776] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 06:09:42 (1789639782)
[ 4838.659909] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 06:10:11 (1789639811)
[ 4840.022321] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 4841.681330] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 06:10:14 (1789639814)
[ 4842.839570] Lustre: *** cfs_fail_loc=805, val=0***
[ 4842.893559] Lustre: Mounted lustre-client - version 2.17.58_39_ga8a99f2
[ 4848.162581] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4868.581963] Lustre: lustre-MDT0000-mdc-ffff9a1e208f5800: Connection to lustre-MDT0000 (at 192.168.204.151@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4868.615217] Lustre: Skipped 12 previous similar messages
[ 4868.618331] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 192.168.204.151@tcp) was lost; in progress operations using this service will fail
[ 4878.839717] LustreError: 2352:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9a1e2080b800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff9a1e0432d600) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4878.888802] Lustre: lustre-MDT0000-mdc-ffff9a1e2080b800: Connection restored to 192.168.204.151@tcp (at 192.168.204.151@tcp)
[ 4878.907563] Lustre: Skipped 12 previous similar messages
[ 4878.929265] Lustre: Evicted from MGS (at 192.168.204.151@tcp) after server handle changed from 0x22220beb02a51bcd to 0x22220beb02a51f7e
[ 4882.134035] Lustre: Unmounted lustre-client
[ 4890.842336] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 06:11:03 (1789639863)
[ 4915.179562] Lustre: 68304:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.151@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 4934.170165] LustreError: 2352:0:(client.c:3449:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff9a1e035bad80 x1876568384645120/t94489280516(94489280516) o101->lustre-MDT0000-mdc-ffff9a1e208f5800@192.168.204.151@tcp:12/10 lens 648/608 e 0 to 0 dl 1789639924 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 4941.253994] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4943.314923] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4953.293324] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 06:12:06 (1789639926)
[ 4954.697798] LustreError: lustre-OST0000-osc-ffff9a1e208f5800: operation ost_write to node 192.168.204.151@tcp failed: rc = -107
[ 4954.710192] LustreError: Skipped 3 previous similar messages
[ 4954.724658] LustreError: lustre-OST0000-osc-ffff9a1e208f5800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4954.753225] Lustre: 2356:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.151@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 4954.812875] LustreError: 72842:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff9a1e208f5800: namespace resource [0x240000400:0x47c2:0x0].0x0 (ffff9a1e04041f00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4954.839992] LustreError: 72842:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4964.026572] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 06:12:16 (1789639936)
[ 4965.604742] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 4967.544840] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 06:12:20 (1789639940)
[ 4969.170751] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 4971.071582] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 06:12:24 (1789639944)
[ 4972.559772] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 4974.188831] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 06:12:27 (1789639947)
[ 4975.911550] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 4978.241325] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 06:12:30 (1789639950)
[ 4979.741568] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 4981.633686] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 06:12:34 (1789639954)
[ 4983.299646] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 4985.015624] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 06:12:38 (1789639958)
[ 4986.403514] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 4988.338493] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 06:12:41 (1789639961)
[ 4990.230843] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 4992.021628] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 06:12:45 (1789639965)
[ 4993.555889] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 4995.718161] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 06:12:48 (1789639968)
[ 4997.680656] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 4999.777878] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 06:12:52 (1789639972)
[ 5001.462989] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5003.560520] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 06:12:56 (1789639976)
[ 5006.704517] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5009.068386] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 06:13:01 (1789639981)
[ 5044.186517] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 06:13:37 (1789640017)
[ 5074.687275] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 06:14:07 (1789640047)
[ 5075.103904] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5085.992709] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 06:14:19 (1789640059)
[ 5086.494418] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5097.751974] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 06:14:30 (1789640070)
[ 5098.263110] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5108.429373] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 06:14:41 (1789640081)
[ 5108.735448] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5118.397924] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 06:14:51 (1789640091)
[ 5118.892480] Lustre: *** cfs_fail_loc=510, val=3***
[ 5128.011530] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 06:15:01 (1789640101)
[ 5128.568209] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5188.064553] Lustre: 2353:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789640107/real 1789640107]  req@ffff9a1e067c8380 x1876568384688896/t0(0) o3->lustre-OST0000-osc-ffff9a1e208f5800@192.168.204.151@tcp:6/4 lens 488/4536 e 0 to 1 dl 1789640162 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 5188.105380] Lustre: 2353:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 13 previous similar messages
[ 5195.935597] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 06:16:09 (1789640169)
[ 5196.457463] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5261.554982] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 06:17:14 (1789640234)
[ 5261.993471] LustreError: 83060:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5265.343309] LustreError: lustre-MDT0000-mdc-ffff9a1e208f5800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5265.371775] LustreError: 83074:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9a1e208f5800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9a1e0432db00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5266.064143] LustreError: 83060:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5268.163335] LustreError: 83090:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5271.066951] LustreError: lustre-MDT0000-mdc-ffff9a1e208f5800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5271.094061] LustreError: 83104:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9a1e208f5800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9a1e0432d000) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5271.134981] LustreError: 83104:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5271.156290] LustreError: 83109:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5271.179117] LustreError: 83109:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5272.168553] LustreError: 83090:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5279.068662] LustreError: lustre-MDT0000-mdc-ffff9a1e208f5800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5279.091829] LustreError: 83133:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9a1e208f5800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff9a1e0432da00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5279.112043] LustreError: 83133:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5279.545365] LustreError: 83146:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5282.585267] LustreError: lustre-MDT0000-mdc-ffff9a1e208f5800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5282.667172] LustreError: 83166:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5283.624202] LustreError: 83146:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5283.640481] LustreError: 83146:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9a1e208f5800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5285.032176] LustreError: 83183:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5285.051964] LustreError: 83183:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 12 previous similar messages
[ 5289.944501] LustreError: 83221:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5292.953604] LustreError: lustre-MDT0000-mdc-ffff9a1e208f5800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5292.970679] LustreError: 83236:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9a1e208f5800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9a1e0a7f7500) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5294.024140] LustreError: 83221:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5300.846204] LustreError: lustre-MDT0000-mdc-ffff9a1e208f5800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5301.179955] LustreError: 83277:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5303.966985] LustreError: lustre-MDT0000-mdc-ffff9a1e208f5800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5305.259206] LustreError: 83277:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5308.732889] LustreError: lustre-MDT0000-mdc-ffff9a1e208f5800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5308.933927] LustreError: 83325:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5308.950421] LustreError: 83325:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5309.521728] LustreError: 83305:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9a1e208f5800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5318.112926] LustreError: 83377:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5318.127109] LustreError: 83377:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5321.810934] LustreError: lustre-MDT0000-mdc-ffff9a1e208f5800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5321.929587] LustreError: 83399:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5321.941977] LustreError: 83399:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5322.161450] LustreError: 83377:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5322.171730] LustreError: 83377:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5322.180407] LustreError: 83377:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9a1e208f5800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5334.665407] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 06:18:27 (1789640307)
[ 5354.343087] LustreError: 68301:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5354.353845] LustreError: 68301:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5358.416122] LustreError: 68301:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5358.424170] LustreError: 68301:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5365.120583] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 06:18:58 (1789640338)
[ 5415.041541] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 06:19:48 (1789640388)
[ 5472.225204] Lustre: lustre-MDT0000-mdc-ffff9a1e208f5800: Connection to lustre-MDT0000 (at 192.168.204.151@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5472.252770] Lustre: Skipped 17 previous similar messages
[ 5479.896683] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 06:20:52 (1789640452)
[ 5513.455578] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 06:21:26 (1789640486)
[ 5513.913947] Lustre: Mounted lustre-client - version 2.17.58_39_ga8a99f2
[ 5637.072099] Lustre: Unmounted lustre-client
[ 5643.586331] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 06:23:36 (1789640616)
[ 5644.215155] LustreError: 87709:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5646.411370] LustreError: lustre-OST0000-osc-ffff9a1e208f5800: operation ldlm_enqueue to node 192.168.204.151@tcp failed: rc = -107
[ 5646.423049] LustreError: Skipped 9 previous similar messages
[ 5646.439117] LustreError: lustre-OST0000-osc-ffff9a1e208f5800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5646.462403] LustreError: 87792:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff9a1e208f5800: namespace resource [0x240000400:0x47ed:0x0].0x0 (ffff9a1e0432da00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5646.477593] LustreError: 87792:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 4 previous similar messages
[ 5646.824173] LustreError: 87709:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5646.832208] Lustre: 2356:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.151@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5652.555555] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 06:23:45 (1789640625)
[ 5709.813516] Lustre: lustre-MDT0000-mdc-ffff9a1e208f5800: Connection restored to 192.168.204.151@tcp (at 192.168.204.151@tcp)
[ 5709.833828] Lustre: Skipped 18 previous similar messages
[ 5717.142117] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 06:24:50 (1789640690)
[ 5718.735757] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 5720.457787] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 06:24:53 (1789640693)
[ 5783.955295] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 5785.545599] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 06:25:58 (1789640758)
[ 5841.888210] Lustre: 90105:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789640761/real 1789640761]  req@ffff9a1e06642680 x1876568384843136/t0(0) o36->lustre-MDT0000-mdc-ffff9a1e208f5800@192.168.204.151@tcp:12/10 lens 496/440 e 0 to 1 dl 1789640816 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ln.0' uid:0 gid:0 projid:4294967295
[ 5841.966244] Lustre: 90105:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
[ 5843.523512] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 06:26:56 (1789640816)
[ 5845.695271] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 5847.718118] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 06:27:00 (1789640820)
[ 5849.000456] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 5851.209906] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 06:27:04 (1789640824)
[ 5887.711393] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 06:27:40 (1789640860)
[ 5899.943193] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5923.744303] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 192.168.204.151@tcp) was lost; in progress operations using this service will fail
[ 5923.770017] LustreError: Skipped 2 previous similar messages
[ 5934.154704] Lustre: Evicted from MGS (at 192.168.204.151@tcp) after server handle changed from 0x22220beb02a52a0c to 0x22220beb02a544bc
[ 5934.164347] Lustre: Skipped 2 previous similar messages
[ 5942.885834] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5944.691792] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5956.303545] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 06:28:49 (1789640929)
[ 5958.593330] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 5960.787458] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 06:28:53 (1789640933)
[ 5975.038236] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 192.168.204.151@tcp) was lost; in progress operations using this service will fail
[ 5975.101111] Lustre: Evicted from MGS (at 192.168.204.151@tcp) after server handle changed from 0x22220beb02a544bc to 0x22220beb02a54652
[ 5990.556816] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 06:29:23 (1789640963)
[ 6042.856782] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 06:30:15 (1789641015)
[ 6106.997926] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6111.027210] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6193.632302] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 192.168.204.151@tcp) was lost; in progress operations using this service will fail
[ 6193.641333] LustreError: Skipped 1 previous similar message
[ 6198.767089] Lustre: lustre-MDT0000-mdc-ffff9a1e208f5800: Connection to lustre-MDT0000 (at 192.168.204.151@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6198.793259] Lustre: Skipped 8 previous similar messages
[ 6204.017254] Lustre: Evicted from MGS (at 192.168.204.151@tcp) after server handle changed from 0x22220beb02a548e4 to 0x22220beb02a55a10
[ 6204.048422] Lustre: Skipped 1 previous similar message
[ 6211.746233] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6213.209339] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6251.466863] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6253.318413] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6397.815958] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 06:36:10 (1789641370)
[ 6410.454126] LustreError: lustre-OST0000-osc-ffff9a1e208f5800: operation ost_setattr to node 192.168.204.151@tcp failed: rc = -107
[ 6410.469095] LustreError: Skipped 51 previous similar messages
[ 6465.048833] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6468.265993] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6721.835993] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 06:41:34 (1789641694)
[ 6869.990395] Lustre: lustre-MDT0000-mdc-ffff9a1e208f5800: Connection to lustre-MDT0000 (at 192.168.204.151@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6869.990565] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 192.168.204.151@tcp) was lost; in progress operations using this service will fail
[ 6870.007308] Lustre: Skipped 2 previous similar messages
[ 6870.016980] LustreError: Skipped 1 previous similar message
[ 6870.044577] Lustre: Evicted from MGS (at 192.168.204.151@tcp) after server handle changed from 0x22220beb02a55b0c to 0x22220beb02b4fe56
[ 6870.053055] Lustre: Skipped 1 previous similar message
[ 6870.082759] Lustre: MGC192.168.204.151@tcp: Connection restored to 192.168.204.151@tcp (at 192.168.204.151@tcp)
[ 6870.092389] Lustre: Skipped 12 previous similar messages
[ 6897.512119] Lustre: 2355:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789641834/real 1789641834]  req@ffff9a1e120b3480 x1876568402511744/t0(0) o400->lustre-MDT0000-mdc-ffff9a1e208f5800@192.168.204.151@tcp:12/10 lens 224/224 e 0 to 1 dl 1789641872 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6897.551402] Lustre: 2355:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 24 previous similar messages
[ 6901.189525] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 06:44:34 (1789641874)
[ 6902.679845] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 6904.280385] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 06:44:37 (1789641877)
[ 6906.704909] LustreError: lustre-MDT0000-mdc-ffff9a1e208f5800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6913.820957] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 06:44:47 (1789641887)
[ 7082.481377] LustreError: lustre-OST0000-osc-ffff9a1e208f5800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7084.270462] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 06:47:37 (1789642057)
[ 7127.986473] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 06:48:21 (1789642101)
[ 7129.673527] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7131.403641] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 06:48:24 (1789642104)
[ 7133.074248] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7135.007652] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 06:48:27 (1789642107)
[ 7168.748751] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 06:49:01 (1789642141)
[ 7207.416838] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 192.168.204.151@tcp) was lost; in progress operations using this service will fail
[ 7207.446439] Lustre: Evicted from MGS (at 192.168.204.151@tcp) after server handle changed from 0x22220beb02b4fe56 to 0x22220beb02b526c0
[ 7207.466024] Lustre: Skipped 1 previous similar message
[ 7223.917995] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 06:49:56 (1789642196)
[ 7225.758394] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7228.054989] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 06:50:00 (1789642200)
[ 7229.562881] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7231.232286] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 06:50:04 (1789642204)
[ 7242.136074] Lustre: Unmounted lustre-client
[ 7244.591855] Lustre: Mounted lustre-client - version 2.17.58_39_ga8a99f2
[ 7247.642040] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7278.208722] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2