[ 1969.414208] LustreError: lustre-OST0000-osc-ffff9a56d02c1800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1969.443284] Lustre: 2354:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.151@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1976.517943] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 12:10:44 (1788624644)
[ 1978.453550] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1980.450719] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 12:10:48 (1788624648)
[ 1982.298611] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1984.297390] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 12:10:51 (1788624651)
[ 1988.121970] LustreError: lustre-MDT0000-mdc-ffff9a56d02c1800: operation mds_close to node 192.168.202.151@tcp failed: rc = -19
[ 1988.138198] LustreError: Skipped 2 previous similar messages
[ 2007.007235] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 192.168.202.151@tcp) was lost; in progress operations using this service will fail
[ 2017.194117] Lustre: Evicted from MGS (at 192.168.202.151@tcp) after server handle changed from 0x895bdb912c5668e0 to 0x895bdb912c5686a0
[ 2113.285320] LustreError: lustre-MDT0000-mdc-ffff9a56d02c1800: operation mds_reint to node 192.168.202.151@tcp failed: rc = -19
[ 2133.983312] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 192.168.202.151@tcp) was lost; in progress operations using this service will fail
[ 2134.017167] Lustre: Evicted from MGS (at 192.168.202.151@tcp) after server handle changed from 0x895bdb912c5686a0 to 0x895bdb912c583693
[ 2135.297793] Lustre: 15925:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.151@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2160.633550] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 12:13:48 (1788624828)
[ 2161.230412] Lustre: *** cfs_fail_loc=305, val=0***
[ 2161.232887] Lustre: Skipped 2 previous similar messages
[ 2180.255402] LustreError: lustre-OST0001-osc-ffff9a56d02c1800: operation ost_connect to node 192.168.202.151@tcp failed: rc = -75
[ 2205.663421] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 192.168.202.151@tcp) was lost; in progress operations using this service will fail
[ 2215.917919] Lustre: Evicted from MGS (at 192.168.202.151@tcp) after server handle changed from 0x895bdb912c583693 to 0x895bdb912c583d9a
[ 2215.919188] Lustre: lustre-MDT0000-mdc-ffff9a56d02c1800: Connection to lustre-MDT0000 (at 192.168.202.151@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2215.934137] Lustre: MGC192.168.202.151@tcp: Connection restored to 192.168.202.151@tcp (at 192.168.202.151@tcp)
[ 2215.940944] Lustre: Skipped 11 previous similar messages
[ 2215.949365] Lustre: Skipped 13 previous similar messages
[ 2228.367541] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2229.943721] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2237.733725] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 12:15:05 (1788624905)
[ 2251.760709] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 192.168.202.151@tcp) was lost; in progress operations using this service will fail
[ 2251.795883] Lustre: Evicted from MGS (at 192.168.202.151@tcp) after server handle changed from 0x895bdb912c583d9a to 0x895bdb912c584216
[ 2251.925944] LustreError: lustre-MDT0000-mdc-ffff9a56d02c1800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2275.542142] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 12:15:43 (1788624943)
[ 2292.718414] LustreError: lustre-OST0000-osc-ffff9a56d02c1800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2309.026691] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 12:16:16 (1788624976)
[ 2322.147336] LustreError: lustre-MDT0000-mdc-ffff9a56d02c1800: operation mds_reint to node 192.168.202.151@tcp failed: rc = -19
[ 2338.785141] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 192.168.202.151@tcp) was lost; in progress operations using this service will fail
[ 2348.968930] Lustre: Evicted from MGS (at 192.168.202.151@tcp) after server handle changed from 0x895bdb912c584216 to 0x895bdb912c5898a8
[ 2350.064653] Lustre: 15925:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.151@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2370.510142] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2373.279905] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2456.543275] Lustre: 2353:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788625109/real 1788625109]  req@ffff9a56d02f8a80 x1875506893541376/t0(0) o400->MGC192.168.202.151@tcp@192.168.202.151@tcp:26/25 lens 224/224 e 0 to 1 dl 1788625125 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2456.591935] Lustre: 2353:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 12 previous similar messages
[ 2456.606167] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 192.168.202.151@tcp) was lost; in progress operations using this service will fail
[ 2465.782353] Lustre: Evicted from MGS (at 192.168.202.151@tcp) after server handle changed from 0x895bdb912c5898a8 to 0x895bdb912c5a6fdf
[ 2467.439588] Lustre: 15925:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.151@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2486.889661] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2489.497902] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2573.280583] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 192.168.202.151@tcp) was lost; in progress operations using this service will fail
[ 2582.703233] Lustre: Evicted from MGS (at 192.168.202.151@tcp) after server handle changed from 0x895bdb912c5a6fdf to 0x895bdb912c5c486d
[ 2594.709816] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2597.196186] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2625.921493] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 12:21:34 (1788625294)
[ 2631.021924] LustreError: lustre-MDT0000-mdc-ffff9a56d02c1800: operation mds_reint to node 192.168.202.151@tcp failed: rc = -19
[ 2631.035821] LustreError: Skipped 3 previous similar messages
[ 2650.143430] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 192.168.202.151@tcp) was lost; in progress operations using this service will fail
[ 2650.169429] Lustre: Evicted from MGS (at 192.168.202.151@tcp) after server handle changed from 0x895bdb912c5c486d to 0x895bdb912c5d366d
[ 2652.078611] Lustre: 15925:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.151@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2669.410702] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2704.134438] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2732.421768] Lustre: 15925:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.151@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2753.248459] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2786.298567] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 192.168.202.151@tcp) was lost; in progress operations using this service will fail
[ 2786.320225] LustreError: Skipped 2 previous similar messages
[ 2786.333260] Lustre: Evicted from MGS (at 192.168.202.151@tcp) after server handle changed from 0x895bdb912c5d94ae to 0x895bdb912c5dff0c
[ 2786.348782] Lustre: Skipped 2 previous similar messages
[ 2796.204288] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2820.643837] Lustre: lustre-MDT0000-mdc-ffff9a56d02c1800: Connection to lustre-MDT0000 (at 192.168.202.151@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2820.660686] Lustre: Skipped 9 previous similar messages
[ 2848.754191] Lustre: MGC192.168.202.151@tcp: Connection restored to 192.168.202.151@tcp (at 192.168.202.151@tcp)
[ 2848.764645] Lustre: Skipped 18 previous similar messages
[ 2858.625627] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2929.104087] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 3025.748883] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 12:28:13 (1788625693)
[ 3081.988932] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3084.253413] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3372.015952] LustreError: lustre-OST0000-osc-ffff9a56d02c1800: operation ldlm_enqueue to node 192.168.202.151@tcp failed: rc = -107
[ 3372.034073] LustreError: Skipped 8 previous similar messages
[ 3410.928969] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3413.376220] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3702.485944] Lustre: lustre-OST0000-osc-ffff9a56d02c1800: Connection to lustre-OST0000 (at 192.168.202.151@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3702.516323] Lustre: Skipped 4 previous similar messages
[ 3724.536228] Lustre: lustre-OST0000-osc-ffff9a56d02c1800: Connection restored to 192.168.202.151@tcp (at 192.168.202.151@tcp)
[ 3724.548790] Lustre: Skipped 7 previous similar messages
[ 3743.243693] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3745.683148] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3998.422874] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 12:44:26 (1788626666)
[ 4016.608618] Lustre: 50579:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788626669/real 1788626669]  req@ffff9a56c4b48000 x1875506917191936/t0(0) o101->lustre-MDT0000-mdc-ffff9a56d02c1800@192.168.202.151@tcp:12/10 lens 576/1152 e 0 to 1 dl 1788626685 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 4016.630912] Lustre: 50579:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
[ 4025.342206] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 12:44:53 (1788626693)
[ 4051.590619] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 12:45:19 (1788626719)
[ 4076.983649] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 12:45:44 (1788626744)
[ 4077.576379] Lustre: Mounted lustre-client - version 2.17.58_2_gf69a9da
[ 4109.215179] Lustre: 2356:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788626762/real 1788626762]  req@ffff9a56c4b49880 x1875506917211136/t0(0) o400->lustre-MDT0000-mdc-ffff9a56d02c1800@192.168.202.151@tcp:12/10 lens 224/224 e 0 to 1 dl 1788626778 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4109.248620] Lustre: 2356:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 4109.279381] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 192.168.202.151@tcp) was lost; in progress operations using this service will fail
[ 4109.288368] LustreError: Skipped 3 previous similar messages
[ 4119.540397] Lustre: Evicted from MGS (at 192.168.202.151@tcp) after server handle changed from 0x895bdb912c60368a to 0x895bdb912c763301
[ 4119.545775] Lustre: Skipped 3 previous similar messages
[ 4125.929726] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4127.329936] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4129.207242] Lustre: Unmounted lustre-client
[ 4135.246373] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 12:46:43 (1788626803)
[ 4272.095212] Lustre: 2353:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788626925/real 1788626925]  req@ffff9a56c76a4380 x1875506917244416/t0(0) o4->lustre-OST0000-osc-ffff9a56d02c1800@192.168.202.151@tcp:6/4 lens 488/448 e 0 to 1 dl 1788626941 ref 2 fl Rpc:XQr/602/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4272.150431] Lustre: 2353:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 83 previous similar messages
[ 4303.839648] Lustre: lustre-OST0000-osc-ffff9a56d02c1800: Connection to lustre-OST0000 (at 192.168.202.151@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4303.870631] Lustre: Skipped 14 previous similar messages
[ 4336.640959] Lustre: lustre-OST0000-osc-ffff9a56d02c1800: Connection restored to 192.168.202.151@tcp (at 192.168.202.151@tcp)
[ 4336.648917] Lustre: Skipped 17 previous similar messages
[ 4429.439970] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 12:51:37 (1788627097)
[ 4479.875890] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 12:52:27 (1788627147)
[ 4483.960993] Lustre: Unmounted lustre-client
[ 4507.566073] Lustre: Mounted lustre-client - version 2.17.58_2_gf69a9da
[ 4516.433597] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 12:53:03 (1788627183)
[ 4517.016122] LustreError: 56510:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4518.064086] LustreError: 56510:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4518.278884] Lustre: *** cfs_fail_loc=305, val=0***
[ 4540.722223] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 12:53:28 (1788627208)
[ 4541.030251] Lustre: Mounted lustre-client - version 2.17.58_2_gf69a9da
[ 4544.303568] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4554.713703] Lustre: Unmounted lustre-client
[ 4562.277095] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 12:53:50 (1788627230)
[ 4642.428049] LustreError: lustre-MDT0000-mdc-ffff9a56c4be8000: operation mds_reint to node 192.168.202.151@tcp failed: rc = -19
[ 4642.435519] LustreError: Skipped 1 previous similar message
[ 4659.999147] Lustre: 2354:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788627313/real 1788627313]  req@ffff9a56c40f0e00 x1875506919921536/t0(0) o400->MGC192.168.202.151@tcp@192.168.202.151@tcp:26/25 lens 224/224 e 0 to 1 dl 1788627329 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4660.027303] Lustre: 2354:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 77 previous similar messages
[ 4660.032430] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 192.168.202.151@tcp) was lost; in progress operations using this service will fail
[ 4670.251397] Lustre: Evicted from MGS (at 192.168.202.151@tcp) after server handle changed from 0x895bdb912c764f94 to 0x895bdb912c7902d4
[ 4822.273474] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 12:58:10 (1788627490)
[ 4826.950655] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4843.515512] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 192.168.202.151@tcp) was lost; in progress operations using this service will fail
[ 4843.546644] Lustre: Evicted from MGS (at 192.168.202.151@tcp) after server handle changed from 0x895bdb912c7902d4 to 0x895bdb912c7dcf7e
[ 4843.607655] LustreError: lustre-MDT0000-mdc-ffff9a56c4be8000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4858.036755] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 12:58:45 (1788627525)
[ 4858.653765] Lustre: Mounted lustre-client - version 2.17.58_2_gf69a9da
[ 4865.972792] LustreError: lustre-OST0000-osc-ffff9a56c4be8000: operation ldlm_enqueue to node 192.168.202.151@tcp failed: rc = -107
[ 4866.002684] LustreError: lustre-OST0000-osc-ffff9a56c4be8000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4866.036549] LustreError: 60230:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff9a56c4be8000: namespace resource [0x240000400:0x4283:0x0].0x0 (ffff9a56c4293a00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4877.989178] Lustre: Unmounted lustre-client
[ 4885.804486] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 12:59:13 (1788627553)
[ 4893.239880] LustreError: lustre-MDT0000-mdc-ffff9a56c4be8000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4893.256246] LustreError: 60900:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4902.645495] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 12:59:30 (1788627570)
[ 4902.986648] LustreError: 61602:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4907.999159] LustreError: 61602:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=0
[ 4908.013545] LustreError: 61602:0:(import.c:755:ptlrpc_connect_import_locked()) already connected
[ 4908.825498] Lustre: lustre-MDT0000-mdc-ffff9a56c4be8000: Connection to lustre-MDT0000 (at 192.168.202.151@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4908.856610] Lustre: Skipped 16 previous similar messages
[ 4908.902472] LustreError: lustre-MDT0000-mdc-ffff9a56c4be8000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4908.928820] LustreError: 61622:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4909.022336] LustreError: 61627:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4910.192614] LustreError: 61633:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4910.216572] LustreError: 61633:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4912.682976] LustreError: 61655:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4912.698853] LustreError: 61655:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4917.564389] LustreError: 61700:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4917.594428] LustreError: 61700:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4929.447861] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 12:59:56 (1788627596)
[ 4971.899716] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4973.655351] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4984.870594] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 13:00:52 (1788627652)
[ 5013.100312] Lustre: lustre-OST0000-osc-ffff9a56c4be8000: Connection restored to 192.168.202.151@tcp (at 192.168.202.151@tcp)
[ 5013.115327] Lustre: Skipped 16 previous similar messages
[ 5031.161363] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5033.203178] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5045.022984] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 13:01:52 (1788627712)
[ 5115.099529] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5116.758025] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5125.844069] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 13:03:13 (1788627793)
[ 5165.806918] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5168.119816] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5174.130175] Lustre: Unmounted lustre-client
[ 5236.307923] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5238.995765] Lustre: Mounted lustre-client - version 2.17.58_2_gf69a9da
[ 5248.839461] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 13:05:16 (1788627916)
[ 5253.055929] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5255.378019] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 13:05:22 (1788627922)
[ 5294.694458] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 13:06:01 (1788627961)
[ 5296.359831] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5298.404864] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 13:06:06 (1788627966)
[ 5300.109462] Lustre: *** cfs_fail_loc=805, val=0***
[ 5300.168472] Lustre: Mounted lustre-client - version 2.17.58_2_gf69a9da
[ 5304.752881] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5325.729327] Lustre: 2354:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788627979/real 1788627979]  req@ffff9a56d1111180 x1875506921978880/t0(0) o400->lustre-MDT0000-mdc-ffff9a56c609d800@192.168.202.151@tcp:12/10 lens 224/224 e 0 to 1 dl 1788627995 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5325.795568] Lustre: 2354:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5325.824428] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 192.168.202.151@tcp) was lost; in progress operations using this service will fail
[ 5336.057432] Lustre: Evicted from MGS (at 192.168.202.151@tcp) after server handle changed from 0x895bdb912c7dd972 to 0x895bdb912c7ddb8d
[ 5337.203493] Lustre: 68285:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.151@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 5351.405933] LustreError: 2352:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9a56c4b64000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff9a56e0050900) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5354.645647] Lustre: Unmounted lustre-client
[ 5364.750366] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 13:07:12 (1788628032)
[ 5392.480845] LustreError: 2352:0:(client.c:3449:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff9a56d1111180 x1875506921991680/t94489280517(94489280517) o101->lustre-MDT0000-mdc-ffff9a56c609d800@192.168.202.151@tcp:12/10 lens 648/608 e 0 to 0 dl 1788628077 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5407.309739] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5409.103943] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5418.313858] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 13:08:06 (1788628086)
[ 5419.586632] LustreError: lustre-OST0000-osc-ffff9a56c609d800: operation ost_write to node 192.168.202.151@tcp failed: rc = -107
[ 5419.606277] LustreError: Skipped 2 previous similar messages
[ 5419.628882] LustreError: lustre-OST0000-osc-ffff9a56c609d800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5419.653240] Lustre: 2355:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.151@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5419.688806] LustreError: 72832:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff9a56c609d800: namespace resource [0x240000400:0x42e2:0x0].0x0 (ffff9a56c4293300) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5419.704540] LustreError: 72832:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5429.253899] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 13:08:17 (1788628097)
[ 5430.802513] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5432.782150] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 13:08:20 (1788628100)
[ 5434.725695] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5436.398886] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 13:08:24 (1788628104)
[ 5437.952833] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5440.005501] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 13:08:28 (1788628108)
[ 5441.768208] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5444.108842] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 13:08:31 (1788628111)
[ 5445.608880] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5447.339695] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 13:08:35 (1788628115)
[ 5448.659800] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5450.839661] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 13:08:38 (1788628118)
[ 5452.599189] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5454.451666] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 13:08:42 (1788628122)
[ 5455.877231] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5457.943655] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 13:08:45 (1788628125)
[ 5459.481858] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5461.434288] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 13:08:49 (1788628129)
[ 5463.098372] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5465.511203] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 13:08:53 (1788628133)
[ 5467.131403] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5468.959306] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 13:08:56 (1788628136)
[ 5472.012848] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5474.156767] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 13:09:01 (1788628141)
[ 5509.546174] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 13:09:37 (1788628177)
[ 5541.658324] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 13:10:09 (1788628209)
[ 5542.146086] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5553.083980] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 13:10:21 (1788628221)
[ 5553.715027] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5565.266445] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 13:10:33 (1788628233)
[ 5565.829223] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5575.163500] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 13:10:43 (1788628243)
[ 5575.719165] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5585.315753] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 13:10:53 (1788628253)
[ 5585.875796] Lustre: *** cfs_fail_loc=510, val=3***
[ 5596.219546] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 13:11:03 (1788628263)
[ 5596.535365] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5655.007508] Lustre: lustre-OST0000-osc-ffff9a56c609d800: Connection to lustre-OST0000 (at 192.168.202.151@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5655.025093] Lustre: Skipped 9 previous similar messages
[ 5663.136642] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 13:12:10 (1788628330)
[ 5663.619266] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5729.920332] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 13:13:18 (1788628398)
[ 5730.846318] LustreError: 83050:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5733.945856] LustreError: lustre-MDT0000-mdc-ffff9a56c609d800: operation ldlm_enqueue to node 192.168.202.151@tcp failed: rc = -107
[ 5733.968836] LustreError: lustre-MDT0000-mdc-ffff9a56c609d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5733.999526] LustreError: 83064:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9a56c609d800: inode [0x20000a042:0xf:0x0] mdc close failed: rc = -108
[ 5734.044828] LustreError: 83064:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9a56c609d800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9a56d4210700) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5734.076593] Lustre: lustre-MDT0000-mdc-ffff9a56c609d800: Connection restored to 192.168.202.151@tcp (at 192.168.202.151@tcp)
[ 5734.084419] Lustre: Skipped 9 previous similar messages
[ 5734.858494] LustreError: 83050:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5737.009476] LustreError: 83080:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5740.039585] LustreError: lustre-MDT0000-mdc-ffff9a56c609d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5740.058492] LustreError: 83094:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9a56c609d800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9a56c83fb800) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5740.074058] LustreError: 83094:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5741.015201] LustreError: 83080:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5748.048277] LustreError: lustre-MDT0000-mdc-ffff9a56c609d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5748.412568] LustreError: 83136:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5751.651666] LustreError: lustre-MDT0000-mdc-ffff9a56c609d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5751.785099] LustreError: 83156:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5751.795258] LustreError: 83156:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5752.487141] LustreError: 83136:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5752.500400] LustreError: 83136:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9a56c609d800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5752.943326] LustreError: 83162:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5752.955214] LustreError: 83162:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5755.314280] LustreError: 83184:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5755.332028] LustreError: 83184:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5759.185091] LustreError: 83211:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5762.352163] LustreError: lustre-MDT0000-mdc-ffff9a56c609d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5762.377207] LustreError: 83226:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9a56c609d800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9a56e0050a00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5762.432030] LustreError: 83226:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5763.283500] LustreError: 83211:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5770.280440] LustreError: lustre-MDT0000-mdc-ffff9a56c609d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5770.600280] LustreError: 83267:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5773.626315] LustreError: lustre-MDT0000-mdc-ffff9a56c609d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5774.679176] LustreError: 83267:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5777.650987] LustreError: lustre-MDT0000-mdc-ffff9a56c609d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5777.725677] LustreError: 83315:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5777.734557] LustreError: 83315:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5778.888605] LustreError: 83295:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9a56c609d800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5786.885750] LustreError: 83367:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5786.898564] LustreError: 83367:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5790.499710] LustreError: lustre-MDT0000-mdc-ffff9a56c609d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5790.561982] LustreError: 83389:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5790.569428] LustreError: 83389:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5790.895253] LustreError: 83367:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5790.905678] LustreError: 83367:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5790.927466] LustreError: 83367:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9a56c609d800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5802.693332] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 13:14:30 (1788628470)
[ 5821.612164] LustreError: 70711:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5821.617421] LustreError: 70711:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5825.679283] LustreError: 70711:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5825.682477] LustreError: 70711:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5832.219695] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 13:15:00 (1788628500)
[ 5883.469497] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 13:15:51 (1788628551)
[ 5940.191955] Lustre: 85545:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788628554/real 1788628554]  req@ffff9a56c4086a00 x1875506922121600/t0(0) o101->lustre-MDT0000-mdc-ffff9a56c609d800@192.168.202.151@tcp:12/10 lens 576/1152 e 0 to 1 dl 1788628609 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'stat.0' uid:0 gid:0 projid:0
[ 5940.228545] Lustre: 85545:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 20 previous similar messages
[ 5947.664361] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 13:16:55 (1788628615)
[ 5982.829601] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 13:17:30 (1788628650)
[ 5983.503965] Lustre: Mounted lustre-client - version 2.17.58_2_gf69a9da
[ 6108.253343] Lustre: Unmounted lustre-client
[ 6115.145247] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 13:19:43 (1788628783)
[ 6115.779956] LustreError: 87692:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 6118.555776] LustreError: lustre-OST0000-osc-ffff9a56c609d800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6118.572418] LustreError: 87775:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff9a56c609d800: namespace resource [0x240000400:0x430d:0x0].0x0 (ffff9a56c4293e00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6118.587673] LustreError: 87775:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 3 previous similar messages
[ 6119.055195] LustreError: 87692:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout interrupted
[ 6119.063064] Lustre: 2354:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.151@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 6127.087897] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 13:19:54 (1788628794)
[ 6192.985877] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 13:21:01 (1788628861)
[ 6194.769176] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 6196.855820] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 13:21:04 (1788628864)
[ 6262.181719] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6264.029608] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 13:22:11 (1788628931)
[ 6320.096487] Lustre: lustre-MDT0000-mdc-ffff9a56c609d800: Connection to lustre-MDT0000 (at 192.168.202.151@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6320.109466] Lustre: Skipped 16 previous similar messages
[ 6322.081620] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 13:23:09 (1788628989)
[ 6324.061100] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6325.739741] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 13:23:13 (1788628993)
[ 6327.682465] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6330.074265] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 13:23:17 (1788628997)
[ 6366.594034] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 13:23:54 (1788629034)
[ 6379.891628] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6408.160164] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 192.168.202.151@tcp) was lost; in progress operations using this service will fail
[ 6408.179919] LustreError: Skipped 2 previous similar messages
[ 6408.217412] Lustre: Evicted from MGS (at 192.168.202.151@tcp) after server handle changed from 0x895bdb912c7de7bf to 0x895bdb912c7e00d2
[ 6408.237632] Lustre: Skipped 2 previous similar messages
[ 6408.247553] Lustre: MGC192.168.202.151@tcp: Connection restored to 192.168.202.151@tcp (at 192.168.202.151@tcp)
[ 6408.262637] Lustre: Skipped 14 previous similar messages
[ 6422.978329] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6424.269908] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6434.818970] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 13:25:02 (1788629102)
[ 6437.207196] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6439.271341] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 13:25:06 (1788629106)
[ 6454.247049] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 192.168.202.151@tcp) was lost; in progress operations using this service will fail
[ 6454.279502] Lustre: Evicted from MGS (at 192.168.202.151@tcp) after server handle changed from 0x895bdb912c7e00d2 to 0x895bdb912c7e048a
[ 6467.510273] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 13:25:35 (1788629135)
[ 6522.592497] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 13:26:30 (1788629190)
[ 6530.921338] LustreError: lustre-OST0000-osc-ffff9a56c609d800: operation ost_setattr to node 192.168.202.151@tcp failed: rc = -19
[ 6530.941448] LustreError: Skipped 9 previous similar messages
[ 6540.321773] Lustre: 2354:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788629154/real 1788629154]  req@ffff9a56c4084000 x1875506922221184/t0(0) o400->lustre-MDT0000-mdc-ffff9a56c609d800@192.168.202.151@tcp:12/10 lens 224/224 e 0 to 1 dl 1788629209 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6540.365903] Lustre: 2354:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 13 previous similar messages
[ 6589.400147] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6594.010867] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6684.138945] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 192.168.202.151@tcp) was lost; in progress operations using this service will fail
[ 6684.150825] LustreError: Skipped 1 previous similar message
[ 6684.160831] Lustre: Evicted from MGS (at 192.168.202.151@tcp) after server handle changed from 0x895bdb912c7e0811 to 0x895bdb912c7e14e4
[ 6684.174413] Lustre: Skipped 1 previous similar message
[ 6685.991031] Lustre: 68285:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.151@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6710.774551] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6712.236922] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6752.266378] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6754.084356] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6894.477157] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 13:32:42 (1788629562)
[ 6960.867839] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6964.300902] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 7216.782158] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 13:38:04 (1788629884)
[ 7360.483636] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 192.168.202.151@tcp) was lost; in progress operations using this service will fail
[ 7360.490719] LustreError: Skipped 1 previous similar message
[ 7360.502276] Lustre: lustre-MDT0000-mdc-ffff9a56c609d800: Connection to lustre-MDT0000 (at 192.168.202.151@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7360.512632] Lustre: Skipped 7 previous similar messages
[ 7360.523521] Lustre: Evicted from MGS (at 192.168.202.151@tcp) after server handle changed from 0x895bdb912c7e1913 to 0x895bdb912c8dbb99
[ 7360.536745] Lustre: Skipped 1 previous similar message
[ 7360.544500] Lustre: MGC192.168.202.151@tcp: Connection restored to 192.168.202.151@tcp (at 192.168.202.151@tcp)
[ 7360.551933] Lustre: Skipped 9 previous similar messages
[ 7388.127316] Lustre: 2354:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788630019/real 1788630019]  req@ffff9a56c425a680 x1875506939892096/t0(0) o400->lustre-MDT0000-mdc-ffff9a56c609d800@192.168.202.151@tcp:12/10 lens 224/224 e 0 to 1 dl 1788630057 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7388.156731] Lustre: 2354:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
[ 7396.776498] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 13:41:04 (1788630064)
[ 7398.671803] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7400.503722] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 13:41:08 (1788630068)
[ 7402.697905] LustreError: lustre-MDT0000-mdc-ffff9a56c609d800: operation ldlm_enqueue to node 192.168.202.151@tcp failed: rc = -107
[ 7402.713118] LustreError: Skipped 303 previous similar messages
[ 7402.724910] LustreError: lustre-MDT0000-mdc-ffff9a56c609d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7411.072468] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 13:41:18 (1788630078)
[ 7578.602667] LustreError: lustre-OST0000-osc-ffff9a56c609d800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7581.111898] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 13:44:08 (1788630248)
[ 7629.877629] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 13:44:56 (1788630296)
[ 7632.098568] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7634.212793] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 13:45:01 (1788630301)
[ 7636.264764] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7638.677487] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 13:45:06 (1788630306)
[ 7676.084889] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 13:45:44 (1788630344)
[ 7714.810990] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 192.168.202.151@tcp) was lost; in progress operations using this service will fail
[ 7714.858254] Lustre: Evicted from MGS (at 192.168.202.151@tcp) after server handle changed from 0x895bdb912c8dbb99 to 0x895bdb912c8de4ea
[ 7714.873593] Lustre: Skipped 1 previous similar message
[ 7736.283570] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 13:46:44 (1788630404)
[ 7738.269065] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7739.795697] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 13:46:48 (1788630408)
[ 7741.917289] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7743.866963] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 13:46:51 (1788630411)
[ 7758.947796] Lustre: Unmounted lustre-client
[ 7761.994565] Lustre: Mounted lustre-client - version 2.17.58_2_gf69a9da
[ 7765.912426] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7790.269392] Lustre: 106828:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.151@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 7808.770364] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2