[ 1659.302955] LustreError: lustre-OST0000-osc-ffff8bb79926b800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1659.320428] Lustre: 2355:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.139@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1665.544477] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 22:07:08 (1789265228)
[ 1667.665276] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1669.450387] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 22:07:12 (1789265232)
[ 1671.205932] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1672.946867] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 22:07:15 (1789265235)
[ 1695.714971] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 192.168.203.139@tcp) was lost; in progress operations using this service will fail
[ 1695.745975] Lustre: Evicted from MGS (at 192.168.203.139@tcp) after server handle changed from 0xa7de85179f04a0ee to 0xa7de85179f04b8a3
[ 1797.866764] LustreError: lustre-MDT0000-mdc-ffff8bb79926b800: operation mds_reint to node 192.168.203.139@tcp failed: rc = -19
[ 1797.876891] LustreError: Skipped 2 previous similar messages
[ 1817.585670] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 192.168.203.139@tcp) was lost; in progress operations using this service will fail
[ 1817.608521] Lustre: Evicted from MGS (at 192.168.203.139@tcp) after server handle changed from 0xa7de85179f04b8a3 to 0xa7de85179f069cfa
[ 1830.022129] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 22:09:53 (1789265393)
[ 1830.446423] Lustre: *** cfs_fail_loc=305, val=0***
[ 1830.451619] Lustre: Skipped 4 previous similar messages
[ 1868.768179] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 192.168.203.139@tcp) was lost; in progress operations using this service will fail
[ 1879.462196] Lustre: Evicted from MGS (at 192.168.203.139@tcp) after server handle changed from 0xa7de85179f069cfa to 0xa7de85179f06c3ea
[ 1883.992328] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 1885.585140] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1894.359086] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 22:10:57 (1789265457)
[ 1910.260598] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 192.168.203.139@tcp) was lost; in progress operations using this service will fail
[ 1910.287921] Lustre: lustre-MDT0000-mdc-ffff8bb79926b800: Connection to lustre-MDT0000 (at 192.168.203.139@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 1910.302673] Lustre: Evicted from MGS (at 192.168.203.139@tcp) after server handle changed from 0xa7de85179f06c3ea to 0xa7de85179f06c683
[ 1910.307019] Lustre: Skipped 13 previous similar messages
[ 1910.329694] Lustre: MGC192.168.203.139@tcp: Connection restored to 192.168.203.139@tcp (at 192.168.203.139@tcp)
[ 1910.336246] Lustre: Skipped 16 previous similar messages
[ 1910.431789] LustreError: lustre-MDT0000-mdc-ffff8bb79926b800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 1929.590136] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 22:11:32 (1789265492)
[ 1960.856427] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 22:12:03 (1789265523)
[ 1974.398720] LustreError: lustre-MDT0000-mdc-ffff8bb79926b800: operation mds_close to node 192.168.203.139@tcp failed: rc = -19
[ 1974.413024] LustreError: Skipped 2 previous similar messages
[ 1993.247213] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 192.168.203.139@tcp) was lost; in progress operations using this service will fail
[ 1993.283358] Lustre: Evicted from MGS (at 192.168.203.139@tcp) after server handle changed from 0xa7de85179f06c683 to 0xa7de85179f0719d4
[ 2018.890620] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2020.641976] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2103.777409] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 192.168.203.139@tcp) was lost; in progress operations using this service will fail
[ 2114.032705] Lustre: Evicted from MGS (at 192.168.203.139@tcp) after server handle changed from 0xa7de85179f0719d4 to 0xa7de85179f08f5e9
[ 2130.983020] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2132.680619] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2198.005358] LustreError: lustre-MDT0000-mdc-ffff8bb79926b800: operation mds_close to node 192.168.203.139@tcp failed: rc = -19
[ 2198.020094] LustreError: Skipped 1 previous similar message
[ 2216.159338] Lustre: 2356:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789265764/real 1789265764]  req@ffff8bb7a0a83b80 x1876178901541248/t0(0) o400->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 224/224 e 0 to 1 dl 1789265780 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2216.190422] Lustre: 2356:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 14 previous similar messages
[ 2216.200250] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 192.168.203.139@tcp) was lost; in progress operations using this service will fail
[ 2226.408941] Lustre: Evicted from MGS (at 192.168.203.139@tcp) after server handle changed from 0xa7de85179f08f5e9 to 0xa7de85179f0ad3b0
[ 2242.819001] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2244.955898] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2273.682933] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 22:17:16 (1789265836)
[ 2297.311735] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 192.168.203.139@tcp) was lost; in progress operations using this service will fail
[ 2306.844748] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2307.497079] Lustre: Evicted from MGS (at 192.168.203.139@tcp) after server handle changed from 0xa7de85179f0ad3b0 to 0xa7de85179f0ba42f
[ 2349.158262] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2383.296718] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2428.601954] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2471.392323] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 192.168.203.139@tcp) was lost; in progress operations using this service will fail
[ 2471.408602] LustreError: Skipped 3 previous similar messages
[ 2481.590655] Lustre: Evicted from MGS (at 192.168.203.139@tcp) after server handle changed from 0xa7de85179f0c7d13 to 0xa7de85179f0ce9f5
[ 2481.602640] Lustre: Skipped 3 previous similar messages
[ 2485.848980] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2513.302263] LustreError: lustre-MDT0000-mdc-ffff8bb79926b800: operation ldlm_enqueue to node 192.168.203.139@tcp failed: rc = -19
[ 2513.313788] LustreError: Skipped 6 previous similar messages
[ 2513.319398] Lustre: lustre-MDT0000-mdc-ffff8bb79926b800: Connection to lustre-MDT0000 (at 192.168.203.139@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2513.337477] Lustre: Skipped 8 previous similar messages
[ 2540.974789] Lustre: MGC192.168.203.139@tcp: Connection restored to 192.168.203.139@tcp (at 192.168.203.139@tcp)
[ 2540.983241] Lustre: Skipped 16 previous similar messages
[ 2547.477298] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2631.907218] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 22:23:15 (1789266195)
[ 2682.099266] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 2683.891351] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3015.163930] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3016.731448] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3310.278055] LustreError: lustre-OST0000-osc-ffff8bb79926b800: operation ldlm_enqueue to node 192.168.203.139@tcp failed: rc = -19
[ 3310.282971] LustreError: Skipped 7 previous similar messages
[ 3310.285232] Lustre: lustre-OST0000-osc-ffff8bb79926b800: Connection to lustre-OST0000 (at 192.168.203.139@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3310.291390] Lustre: Skipped 3 previous similar messages
[ 3331.088728] Lustre: lustre-OST0000-osc-ffff8bb79926b800: Connection restored to 192.168.203.139@tcp (at 192.168.203.139@tcp)
[ 3331.107254] Lustre: Skipped 4 previous similar messages
[ 3349.159549] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3350.963454] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3605.765742] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 22:39:28 (1789267168)
[ 3622.879968] Lustre: 50572:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789267171/real 1789267171]  req@ffff8bb798a04700 x1876178922427136/t0(0) o101->lustre-MDT0000-mdc-ffff8bb79926b800@192.168.203.139@tcp:12/10 lens 576/1152 e 0 to 1 dl 1789267187 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3622.901556] Lustre: 50572:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
[ 3631.185154] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 22:39:54 (1789267194)
[ 3657.248383] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 22:40:20 (1789267220)
[ 3682.381981] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 22:40:45 (1789267245)
[ 3682.917106] Lustre: Mounted lustre-client - version 2.17.57_103_g3c4c45f
[ 3714.527211] Lustre: 2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789267262/real 1789267262]  req@ffff8bb7a0906d80 x1876178922446976/t0(0) o400->lustre-MDT0000-mdc-ffff8bb799181800@192.168.203.139@tcp:12/10 lens 224/224 e 0 to 1 dl 1789267278 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3714.549651] Lustre: 2355:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 3714.560162] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 192.168.203.139@tcp) was lost; in progress operations using this service will fail
[ 3714.572324] LustreError: Skipped 2 previous similar messages
[ 3714.597784] Lustre: Evicted from MGS (at 192.168.203.139@tcp) after server handle changed from 0xa7de85179f0e5404 to 0xa7de85179f235e29
[ 3714.629166] Lustre: Skipped 2 previous similar messages
[ 3716.592564] Lustre: 15906:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.139@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 3740.745423] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 3742.787709] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3745.144502] Lustre: Unmounted lustre-client
[ 3751.431713] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 22:41:54 (1789267314)
[ 3871.711261] Lustre: 2358:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789267420/real 1789267420]  req@ffff8bb798a69880 x1876178922482432/t0(0) o4->lustre-OST0000-osc-ffff8bb79926b800@192.168.203.139@tcp:6/4 lens 488/448 e 0 to 1 dl 1789267436 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 3871.758771] Lustre: 2358:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 56 previous similar messages
[ 3919.839244] Lustre: lustre-OST0000-osc-ffff8bb79926b800: Connection to lustre-OST0000 (at 192.168.203.139@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3919.875096] Lustre: Skipped 14 previous similar messages
[ 3936.269074] Lustre: lustre-OST0000-osc-ffff8bb79926b800: Connection restored to 192.168.203.139@tcp (at 192.168.203.139@tcp)
[ 3936.276846] Lustre: Skipped 16 previous similar messages
[ 4045.408606] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 22:46:48 (1789267608)
[ 4094.164683] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 22:47:37 (1789267657)
[ 4098.154380] Lustre: Unmounted lustre-client
[ 4119.911807] Lustre: Mounted lustre-client - version 2.17.57_103_g3c4c45f
[ 4125.212915] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 22:48:08 (1789267688)
[ 4125.625423] LustreError: 56499:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4126.688194] LustreError: 56499:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4126.895346] Lustre: *** cfs_fail_loc=305, val=0***
[ 4149.033992] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 22:48:32 (1789267712)
[ 4149.510759] Lustre: Mounted lustre-client - version 2.17.57_103_g3c4c45f
[ 4151.824435] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4162.148197] Lustre: Unmounted lustre-client
[ 4169.666204] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 22:48:52 (1789267732)
[ 4254.659561] LustreError: lustre-MDT0000-mdc-ffff8bb799180000: operation mds_reint to node 192.168.203.139@tcp failed: rc = -19
[ 4272.610947] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 192.168.203.139@tcp) was lost; in progress operations using this service will fail
[ 4272.631536] Lustre: Evicted from MGS (at 192.168.203.139@tcp) after server handle changed from 0xa7de85179f237ba3 to 0xa7de85179f26360d
[ 4414.389401] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 22:52:57 (1789267977)
[ 4419.850555] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4436.470787] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 192.168.203.139@tcp) was lost; in progress operations using this service will fail
[ 4436.499089] Lustre: Evicted from MGS (at 192.168.203.139@tcp) after server handle changed from 0xa7de85179f26360d to 0xa7de85179f2af72d
[ 4436.538464] LustreError: lustre-MDT0000-mdc-ffff8bb799180000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4448.876340] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 22:53:31 (1789268011)
[ 4449.444393] Lustre: Mounted lustre-client - version 2.17.57_103_g3c4c45f
[ 4456.878915] LustreError: lustre-OST0000-osc-ffff8bb799180000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4467.167136] Lustre: 2358:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789268015/real 1789268015]  req@ffff8bb787a18380 x1876178927132928/t0(0) o103->lustre-OST0000-osc-ffff8bb79926f000@192.168.203.139@tcp:17/18 lens 328/224 e 0 to 1 dl 1789268031 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295
[ 4467.199617] Lustre: 2358:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 84 previous similar messages
[ 4469.362705] Lustre: Unmounted lustre-client
[ 4475.451325] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 22:53:58 (1789268038)
[ 4481.818204] LustreError: lustre-MDT0000-mdc-ffff8bb799180000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4481.832566] LustreError: 60885:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4489.082943] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 22:54:12 (1789268052)
[ 4489.236574] LustreError: 61586:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4494.305432] LustreError: 61586:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=0
[ 4494.323790] LustreError: 61586:0:(import.c:711:ptlrpc_connect_import_locked()) already connected
[ 4494.755578] LustreError: lustre-MDT0000-mdc-ffff8bb799180000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4494.774238] LustreError: 61606:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4494.840488] LustreError: 61612:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4495.959281] LustreError: 61618:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4495.965469] LustreError: 61618:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4498.282179] LustreError: 61640:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4498.308186] LustreError: 61640:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4503.012656] LustreError: 61685:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4503.028386] LustreError: 61685:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4513.394646] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 22:54:36 (1789268076)
[ 4520.426081] Lustre: lustre-OST0000-osc-ffff8bb799180000: Connection to lustre-OST0000 (at 192.168.203.139@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4520.443182] Lustre: Skipped 18 previous similar messages
[ 4555.478142] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4557.463417] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4567.597381] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 22:55:30 (1789268130)
[ 4593.264529] Lustre: lustre-OST0000-osc-ffff8bb799180000: Connection restored to 192.168.203.139@tcp (at 192.168.203.139@tcp)
[ 4593.290463] Lustre: Skipped 18 previous similar messages
[ 4608.189066] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4609.829288] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4618.537411] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 22:56:21 (1789268181)
[ 4681.810809] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4683.396914] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4691.309894] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 22:57:34 (1789268254)
[ 4727.365845] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4728.742515] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4734.231403] Lustre: Unmounted lustre-client
[ 4785.665857] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4787.970494] Lustre: Mounted lustre-client - version 2.17.57_103_g3c4c45f
[ 4794.641447] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 22:59:17 (1789268357)
[ 4797.071622] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 4798.658230] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 22:59:21 (1789268361)
[ 4827.228339] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 22:59:50 (1789268390)
[ 4828.587552] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 4830.353296] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 22:59:53 (1789268393)
[ 4831.750964] Lustre: *** cfs_fail_loc=805, val=0***
[ 4831.799306] Lustre: Mounted lustre-client - version 2.17.57_103_g3c4c45f
[ 4836.643575] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4857.318902] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 192.168.203.139@tcp) was lost; in progress operations using this service will fail
[ 4857.338521] Lustre: Evicted from MGS (at 192.168.203.139@tcp) after server handle changed from 0xa7de85179f2b011a to 0xa7de85179f2b0335
[ 4858.992257] Lustre: 68253:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.139@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 4875.050831] Lustre: Unmounted lustre-client
[ 4883.384598] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 23:00:46 (1789268446)
[ 4910.095775] LustreError: 2354:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff8bb798b22a00 x1876178927216256/t90194313220(90194313220) o101->lustre-MDT0000-mdc-ffff8bb79948d000@192.168.203.139@tcp:12/10 lens 664/608 e 0 to 0 dl 1789268490 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 4917.172855] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4918.410320] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4925.371817] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 23:01:28 (1789268488)
[ 4926.720992] LustreError: lustre-OST0000-osc-ffff8bb79948d000: operation ost_write to node 192.168.203.139@tcp failed: rc = -107
[ 4926.729490] LustreError: Skipped 3 previous similar messages
[ 4926.748805] LustreError: lustre-OST0000-osc-ffff8bb79948d000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4926.764823] Lustre: 2356:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.139@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 4926.796139] LustreError: 72805:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff8bb79948d000: namespace resource [0x240000400:0x4082:0x0].0x0 (ffff8bb7906b2000) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4935.899737] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 23:01:38 (1789268498)
[ 4937.404889] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 4939.271748] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 23:01:42 (1789268502)
[ 4940.877590] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 4943.306653] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 23:01:45 (1789268505)
[ 4945.073835] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 4947.377863] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 23:01:49 (1789268509)
[ 4949.087906] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 4951.038493] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 23:01:53 (1789268513)
[ 4952.782519] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 4954.321350] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 23:01:57 (1789268517)
[ 4956.241659] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 4958.053425] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 23:02:00 (1789268520)
[ 4959.866815] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 4961.364471] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 23:02:04 (1789268524)
[ 4962.900955] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 4964.742700] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 23:02:07 (1789268527)
[ 4966.487050] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 4967.798200] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 23:02:11 (1789268531)
[ 4969.041110] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 4970.998705] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 23:02:13 (1789268533)
[ 4972.494754] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 4974.496457] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 23:02:17 (1789268537)
[ 4977.111203] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 4978.744776] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 23:02:21 (1789268541)
[ 5002.240973] Lustre: 68253:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.139@tcp: IR log lustre-cliir failed, not fatal: rc = -5
[ 5010.513669] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 23:02:53 (1789268573)
[ 5039.912620] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 23:03:23 (1789268603)
[ 5040.326795] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5050.776748] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 23:03:33 (1789268613)
[ 5051.052537] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5061.719365] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 23:03:44 (1789268624)
[ 5062.132635] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5071.465720] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 23:03:54 (1789268634)
[ 5071.875368] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5080.982828] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 23:04:03 (1789268643)
[ 5081.284203] Lustre: *** cfs_fail_loc=510, val=3***
[ 5088.898942] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 23:04:12 (1789268652)
[ 5089.100492] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5100.511162] Lustre: 2358:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789268648/real 1789268648]  req@ffff8bb7a007a300 x1876178927255424/t0(0) o400->lustre-OST0000-osc-ffff8bb79948d000@192.168.203.139@tcp:28/4 lens 224/224 e 0 to 1 dl 1789268664 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5100.544215] Lustre: 2358:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 13 previous similar messages
[ 5148.639270] Lustre: lustre-OST0000-osc-ffff8bb79948d000: Connection to lustre-OST0000 (at 192.168.203.139@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5148.655958] Lustre: Skipped 9 previous similar messages
[ 5155.051480] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 23:05:18 (1789268718)
[ 5155.401838] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5222.112865] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 23:06:25 (1789268785)
[ 5222.373972] LustreError: 83021:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5225.121729] LustreError: lustre-MDT0000-mdc-ffff8bb79948d000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5225.154562] LustreError: 83036:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8bb79948d000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8bb798b06c00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5225.187535] Lustre: lustre-MDT0000-mdc-ffff8bb79948d000: Connection restored to 192.168.203.139@tcp (at 192.168.203.139@tcp)
[ 5225.202832] Lustre: Skipped 11 previous similar messages
[ 5226.463273] LustreError: 83021:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5228.586684] LustreError: 83052:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5231.271921] LustreError: lustre-MDT0000-mdc-ffff8bb79948d000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5231.288026] LustreError: 83066:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8bb79948d000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8bb7906b2d00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5231.313270] LustreError: 83066:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 2 previous similar messages
[ 5232.671141] LustreError: 83052:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5239.665900] LustreError: lustre-MDT0000-mdc-ffff8bb79948d000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5240.040226] LustreError: 83108:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5243.125167] LustreError: lustre-MDT0000-mdc-ffff8bb79948d000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5243.223600] LustreError: 83127:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5243.245775] LustreError: 83127:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5244.119123] LustreError: 83108:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5244.125877] LustreError: 83108:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8bb79948d000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5244.353784] LustreError: 83134:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5244.370377] LustreError: 83134:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5246.819785] LustreError: 83156:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5246.832830] LustreError: 83156:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5250.699923] LustreError: 83183:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5253.668781] LustreError: lustre-MDT0000-mdc-ffff8bb79948d000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5253.700509] LustreError: 83197:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8bb79948d000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff8bb79074d400) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5253.729081] LustreError: 83197:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5254.767128] LustreError: 83183:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5262.008989] LustreError: lustre-MDT0000-mdc-ffff8bb79948d000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5262.424884] LustreError: 83241:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5265.386223] LustreError: lustre-MDT0000-mdc-ffff8bb79948d000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5265.417094] LustreError: 83256:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8bb79948d000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8bb7a0301500) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5265.460184] LustreError: 83256:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 2 previous similar messages
[ 5266.495131] LustreError: 83241:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5269.831347] LustreError: lustre-MDT0000-mdc-ffff8bb79948d000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5269.937470] LustreError: 83289:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5269.948344] LustreError: 83289:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5270.832343] LustreError: 83269:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8bb79948d000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5279.082981] LustreError: 83341:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5279.097901] LustreError: 83341:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5282.537719] LustreError: lustre-MDT0000-mdc-ffff8bb79948d000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5282.632956] LustreError: 83363:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5282.640495] LustreError: 83363:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5283.087137] LustreError: 83341:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5283.098256] LustreError: 83341:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5283.112072] LustreError: 83341:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8bb79948d000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5293.878695] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 23:07:37 (1789268857)
[ 5313.082572] LustreError: 68251:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5313.090086] LustreError: 68251:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5317.103735] LustreError: 68251:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5317.123574] LustreError: 68251:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5325.037437] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 23:08:08 (1789268888)
[ 5374.583202] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 23:08:57 (1789268937)
[ 5437.055402] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 23:10:00 (1789269000)
[ 5468.890974] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 23:10:32 (1789269032)
[ 5469.372580] Lustre: Mounted lustre-client - version 2.17.57_103_g3c4c45f
[ 5592.250649] Lustre: Unmounted lustre-client
[ 5598.639157] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 23:12:41 (1789269161)
[ 5599.208642] LustreError: 87666:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5601.556202] LustreError: lustre-OST0000-osc-ffff8bb79948d000: operation ldlm_enqueue to node 192.168.203.139@tcp failed: rc = -107
[ 5601.567107] LustreError: Skipped 9 previous similar messages
[ 5601.583440] LustreError: lustre-OST0000-osc-ffff8bb79948d000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5601.608854] LustreError: 87748:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff8bb79948d000: namespace resource [0x240000400:0x40ad:0x0].0x0 (ffff8bb79074df00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5601.655496] LustreError: 87748:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5601.951134] LustreError: 87666:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5601.955065] Lustre: 2357:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.139@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5607.214815] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 23:12:50 (1789269170)
[ 5671.690297] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 23:13:54 (1789269234)
[ 5673.130989] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 5674.556814] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 23:13:57 (1789269237)
[ 5731.300286] Lustre: 89328:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789269240/real 1789269240]  req@ffff8bb7a007b800 x1876178927400832/t0(0) o101->lustre-MDT0000-mdc-ffff8bb79948d000@192.168.203.139@tcp:12/10 lens 648/66320 e 0 to 1 dl 1789269295 ref 2 fl Rpc:XPQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 5731.324991] Lustre: 89328:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
[ 5736.558023] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 5738.003750] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 23:15:01 (1789269301)
[ 5793.759207] Lustre: lustre-MDT0000-mdc-ffff8bb79948d000: Connection to lustre-MDT0000 (at 192.168.203.139@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5793.780320] Lustre: Skipped 16 previous similar messages
[ 5795.340583] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 23:15:58 (1789269358)
[ 5797.895291] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 5799.934409] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 23:16:02 (1789269362)
[ 5802.125665] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 5805.040795] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 23:16:07 (1789269367)
[ 5844.759784] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 23:16:47 (1789269407)
[ 5855.667736] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5880.799180] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 192.168.203.139@tcp) was lost; in progress operations using this service will fail
[ 5880.815981] LustreError: Skipped 3 previous similar messages
[ 5886.079206] Lustre: lustre-MDT0000-mdc-ffff8bb79948d000: Connection restored to 192.168.203.139@tcp (at 192.168.203.139@tcp)
[ 5886.089521] Lustre: Skipped 14 previous similar messages
[ 5891.128864] Lustre: Evicted from MGS (at 192.168.203.139@tcp) after server handle changed from 0xa7de85179f2b0fa6 to 0xa7de85179f2b2a1e
[ 5891.149309] Lustre: Skipped 3 previous similar messages
[ 5896.963732] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5898.427832] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5908.022543] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 23:17:51 (1789269471)
[ 5910.537114] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 5912.164226] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 23:17:55 (1789269475)
[ 5926.892155] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 192.168.203.139@tcp) was lost; in progress operations using this service will fail
[ 5926.924657] Lustre: Evicted from MGS (at 192.168.203.139@tcp) after server handle changed from 0xa7de85179f2b2a1e to 0xa7de85179f2b2c39
[ 5936.026517] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 23:18:19 (1789269499)
[ 5981.074540] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 23:19:04 (1789269544)
[ 6039.527595] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6042.226983] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6127.391483] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 192.168.203.139@tcp) was lost; in progress operations using this service will fail
[ 6127.402492] LustreError: Skipped 1 previous similar message
[ 6127.424190] Lustre: Evicted from MGS (at 192.168.203.139@tcp) after server handle changed from 0xa7de85179f2b2fc0 to 0xa7de85179f2b3ca1
[ 6127.430272] Lustre: Skipped 1 previous similar message
[ 6141.868983] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6143.373951] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6174.893607] Lustre: 68253:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.139@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6191.361826] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6192.636552] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6324.309646] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 23:24:47 (1789269887)
[ 6336.017546] LustreError: lustre-OST0000-osc-ffff8bb79948d000: operation ost_setattr to node 192.168.203.139@tcp failed: rc = -19
[ 6336.029202] LustreError: Skipped 38 previous similar messages
[ 6390.175726] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6392.983525] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6644.658622] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 23:30:07 (1789270207)
[ 6845.425393] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 192.168.203.139@tcp) was lost; in progress operations using this service will fail
[ 6845.438221] LustreError: Skipped 1 previous similar message
[ 6845.443909] Lustre: lustre-MDT0000-mdc-ffff8bb79948d000: Connection to lustre-MDT0000 (at 192.168.203.139@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6845.445793] Lustre: Evicted from MGS (at 192.168.203.139@tcp) after server handle changed from 0xa7de85179f2b409f to 0xa7de85179f3a763c
[ 6845.454796] Lustre: Skipped 7 previous similar messages
[ 6845.474049] Lustre: Skipped 1 previous similar message
[ 6845.478670] Lustre: MGC192.168.203.139@tcp: Connection restored to 192.168.203.139@tcp (at 192.168.203.139@tcp)
[ 6845.487735] Lustre: Skipped 9 previous similar messages
[ 6873.055661] Lustre: 2358:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789270399/real 1789270399]  req@ffff8bb7a11abb80 x1876178944561408/t0(0) o400->lustre-MDT0000-mdc-ffff8bb79948d000@192.168.203.139@tcp:12/10 lens 224/224 e 0 to 1 dl 1789270437 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6873.089530] Lustre: 2358:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 28 previous similar messages
[ 6873.676161] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 23:33:57 (1789270437)
[ 6874.839918] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 6876.212453] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 23:33:59 (1789270439)
[ 6877.852528] LustreError: lustre-MDT0000-mdc-ffff8bb79948d000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6877.862347] LustreError: 102017:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8bb79948d000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff8bb7b8dbcf00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6877.886986] LustreError: 102017:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 6883.842471] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 23:34:07 (1789270447)
[ 7051.563510] LustreError: lustre-OST0000-osc-ffff8bb79948d000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7053.053491] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 23:36:56 (1789270616)
[ 7095.654883] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 23:37:38 (1789270658)
[ 7097.105685] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7098.747248] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 23:37:41 (1789270661)
[ 7100.176645] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7101.870393] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 23:37:44 (1789270664)
[ 7135.536174] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 23:38:18 (1789270698)
[ 7174.631465] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 192.168.203.139@tcp) was lost; in progress operations using this service will fail
[ 7174.652417] Lustre: Evicted from MGS (at 192.168.203.139@tcp) after server handle changed from 0xa7de85179f3a763c to 0xa7de85179f3a9fef
[ 7174.669326] Lustre: Skipped 1 previous similar message
[ 7197.304509] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 23:39:19 (1789270759)
[ 7199.765674] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7202.531021] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 23:39:24 (1789270764)
[ 7204.480541] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7205.840148] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 23:39:29 (1789270769)
[ 7215.115074] Lustre: Unmounted lustre-client
[ 7216.268679] Lustre: Mounted lustre-client - version 2.17.57_103_g3c4c45f
[ 7219.296695] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7249.986171] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2