[ 1895.383776] LustreError: lustre-OST0000-osc-ffff92e1ca39e800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1895.411407] Lustre: 2350:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.160@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1903.716399] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 07:27:21 (1787830041)
[ 1906.475360] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1909.154727] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 07:27:27 (1787830047)
[ 1911.724529] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1916.997385] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 07:27:35 (1787830055)
[ 1921.703408] LustreError: lustre-MDT0000-mdc-ffff92e1ca39e800: operation mds_reint to node 192.168.204.160@tcp failed: rc = -19
[ 1921.715711] LustreError: Skipped 1 previous similar message
[ 1940.960534] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 192.168.204.160@tcp) was lost; in progress operations using this service will fail
[ 1951.213182] Lustre: Evicted from MGS (at 192.168.204.160@tcp) after server handle changed from 0x40546b8e5040c21c to 0x40546b8e5040db60
[ 2055.214397] LustreError: lustre-MDT0000-mdc-ffff92e1ca39e800: operation mds_reint to node 192.168.204.160@tcp failed: rc = -19
[ 2072.991191] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 192.168.204.160@tcp) was lost; in progress operations using this service will fail
[ 2083.248929] Lustre: Evicted from MGS (at 192.168.204.160@tcp) after server handle changed from 0x40546b8e5040db60 to 0x40546b8e50428200
[ 2085.209762] Lustre: 15949:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.160@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2102.764565] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 07:30:41 (1787830241)
[ 2103.180411] Lustre: *** cfs_fail_loc=305, val=0***
[ 2103.188469] Lustre: Skipped 2 previous similar messages
[ 2148.895820] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 192.168.204.160@tcp) was lost; in progress operations using this service will fail
[ 2158.065707] Lustre: Evicted from MGS (at 192.168.204.160@tcp) after server handle changed from 0x40546b8e50428200 to 0x40546b8e50429937
[ 2158.091233] Lustre: MGC192.168.204.160@tcp: Connection restored to 192.168.204.160@tcp (at 192.168.204.160@tcp)
[ 2158.111068] Lustre: Skipped 13 previous similar messages
[ 2159.990844] Lustre: lustre-MDT0000-mdc-ffff92e1ca39e800: Connection to lustre-MDT0000 (at 192.168.204.160@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2160.016138] Lustre: Skipped 11 previous similar messages
[ 2177.040627] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2178.997414] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2188.565589] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 07:32:07 (1787830327)
[ 2204.146993] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 192.168.204.160@tcp) was lost; in progress operations using this service will fail
[ 2204.180100] Lustre: Evicted from MGS (at 192.168.204.160@tcp) after server handle changed from 0x40546b8e50429937 to 0x40546b8e50429ee0
[ 2204.252314] LustreError: lustre-MDT0000-mdc-ffff92e1ca39e800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2230.346647] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 07:32:48 (1787830368)
[ 2267.561060] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 07:33:25 (1787830405)
[ 2281.882507] LustreError: lustre-MDT0000-mdc-ffff92e1ca39e800: operation mds_reint to node 192.168.204.160@tcp failed: rc = -19
[ 2281.893200] LustreError: Skipped 1 previous similar message
[ 2302.431432] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 192.168.204.160@tcp) was lost; in progress operations using this service will fail
[ 2302.461837] Lustre: Evicted from MGS (at 192.168.204.160@tcp) after server handle changed from 0x40546b8e50429ee0 to 0x40546b8e5042e2f6
[ 2304.086990] Lustre: 15949:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.160@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2332.687903] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2336.259159] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2419.167157] Lustre: 2348:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787830543/real 1787830543]  req@ffff92e1cbb6b840 x1874673733398272/t0(0) o400->MGC192.168.204.160@tcp@192.168.204.160@tcp:26/25 lens 224/224 e 0 to 1 dl 1787830559 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2419.192715] Lustre: 2348:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 18 previous similar messages
[ 2419.197900] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 192.168.204.160@tcp) was lost; in progress operations using this service will fail
[ 2429.429040] Lustre: Evicted from MGS (at 192.168.204.160@tcp) after server handle changed from 0x40546b8e5042e2f6 to 0x40546b8e50443ef0
[ 2430.524930] Lustre: 15949:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.160@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2456.334670] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2458.601207] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2541.023699] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 192.168.204.160@tcp) was lost; in progress operations using this service will fail
[ 2551.220162] Lustre: Evicted from MGS (at 192.168.204.160@tcp) after server handle changed from 0x40546b8e50443ef0 to 0x40546b8e5045b11e
[ 2568.593440] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2571.132666] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2601.691109] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 07:39:00 (1787830740)
[ 2607.319611] LustreError: lustre-MDT0000-mdc-ffff92e1ca39e800: operation mds_close to node 192.168.204.160@tcp failed: rc = -19
[ 2607.336048] LustreError: Skipped 6 previous similar messages
[ 2627.039300] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 192.168.204.160@tcp) was lost; in progress operations using this service will fail
[ 2637.292614] Lustre: Evicted from MGS (at 192.168.204.160@tcp) after server handle changed from 0x40546b8e5045b11e to 0x40546b8e504666ed
[ 2650.172810] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2685.167510] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2732.556880] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2762.207276] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 192.168.204.160@tcp) was lost; in progress operations using this service will fail
[ 2762.218079] LustreError: Skipped 2 previous similar messages
[ 2772.402083] Lustre: Evicted from MGS (at 192.168.204.160@tcp) after server handle changed from 0x40546b8e50469611 to 0x40546b8e5046eadc
[ 2772.410719] Lustre: Skipped 2 previous similar messages
[ 2772.417788] Lustre: MGC192.168.204.160@tcp: Connection restored to 192.168.204.160@tcp (at 192.168.204.160@tcp)
[ 2772.425719] Lustre: Skipped 15 previous similar messages
[ 2784.911465] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2808.529806] Lustre: lustre-MDT0000-mdc-ffff92e1ca39e800: Connection to lustre-MDT0000 (at 192.168.204.160@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2808.544713] Lustre: Skipped 9 previous similar messages
[ 2830.127110] Lustre: 15949:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.160@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2845.121428] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2913.602865] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 3008.204921] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 07:45:47 (1787831147)
[ 3062.166974] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3064.836639] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3352.274887] LustreError: lustre-OST0000-osc-ffff92e1ca39e800: operation ldlm_enqueue to node 192.168.204.160@tcp failed: rc = -19
[ 3352.296986] LustreError: Skipped 10 previous similar messages
[ 3373.220699] Lustre: lustre-OST0000-osc-ffff92e1ca39e800: Connection restored to 192.168.204.160@tcp (at 192.168.204.160@tcp)
[ 3373.233495] Lustre: Skipped 7 previous similar messages
[ 3391.590481] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3394.203472] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3686.382775] Lustre: lustre-OST0000-osc-ffff92e1ca39e800: Connection to lustre-OST0000 (at 192.168.204.160@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3686.395666] Lustre: Skipped 4 previous similar messages
[ 3724.990445] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3726.907231] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3983.108636] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 08:02:01 (1787832121)
[ 4000.223440] Lustre: 50585:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787832124/real 1787832124]  req@ffff92e1d1f2e580 x1874673753874688/t0(0) o101->lustre-MDT0000-mdc-ffff92e1ca39e800@192.168.204.160@tcp:12/10 lens 576/1152 e 0 to 1 dl 1787832140 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 4000.262722] Lustre: 50585:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
[ 4000.316957] Lustre: lustre-MDT0000-mdc-ffff92e1ca39e800: Connection restored to 192.168.204.160@tcp (at 192.168.204.160@tcp)
[ 4000.324622] Lustre: Skipped 1 previous similar message
[ 4009.779479] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 08:02:28 (1787832148)
[ 4040.007335] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 08:02:57 (1787832177)
[ 4070.037102] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 08:03:27 (1787832207)
[ 4070.875303] Lustre: Mounted lustre-client - version 2.17.57_65_g89f8e4a
[ 4102.625365] Lustre: 2350:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787832226/real 1787832226]  req@ffff92e1d0c9cb40 x1874673753895552/t0(0) o400->lustre-MDT0000-mdc-ffff92e1c3e23000@192.168.204.160@tcp:12/10 lens 224/224 e 0 to 1 dl 1787832242 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4102.673831] Lustre: 2350:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 4102.689492] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 192.168.204.160@tcp) was lost; in progress operations using this service will fail
[ 4102.708755] LustreError: Skipped 3 previous similar messages
[ 4112.886135] Lustre: Evicted from MGS (at 192.168.204.160@tcp) after server handle changed from 0x40546b8e5048accd to 0x40546b8e505c448c
[ 4112.898311] Lustre: Skipped 3 previous similar messages
[ 4126.671558] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4128.745445] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4131.298872] Lustre: Unmounted lustre-client
[ 4138.939541] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 08:04:37 (1787832277)
[ 4262.176934] Lustre: 2349:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787832386/real 1787832386]  req@ffff92e1d1f2fc00 x1874673753930496/t0(0) o4->lustre-OST0000-osc-ffff92e1ca39e800@192.168.204.160@tcp:6/4 lens 488/448 e 0 to 1 dl 1787832402 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4262.237482] Lustre: 2349:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 52 previous similar messages
[ 4294.111907] Lustre: lustre-OST0000-osc-ffff92e1ca39e800: Connection to lustre-OST0000 (at 192.168.204.160@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4294.132126] Lustre: Skipped 13 previous similar messages
[ 4439.592595] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 08:09:38 (1787832578)
[ 4494.473196] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 08:10:31 (1787832631)
[ 4498.023309] Lustre: Unmounted lustre-client
[ 4527.342096] Lustre: Mounted lustre-client - version 2.17.57_65_g89f8e4a
[ 4534.757582] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 08:11:13 (1787832673)
[ 4535.102445] LustreError: 56519:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4536.152197] LustreError: 56519:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4536.452276] Lustre: *** cfs_fail_loc=305, val=0***
[ 4562.464065] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 08:11:40 (1787832700)
[ 4563.340247] Lustre: Mounted lustre-client - version 2.17.57_65_g89f8e4a
[ 4567.284968] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4577.617874] Lustre: Unmounted lustre-client
[ 4586.384709] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 08:12:04 (1787832724)
[ 4688.624324] LustreError: lustre-MDT0000-mdc-ffff92e1c51eb800: operation mds_reint to node 192.168.204.160@tcp failed: rc = -19
[ 4688.638331] LustreError: Skipped 1 previous similar message
[ 4708.833300] Lustre: 2350:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787832832/real 1787832832]  req@ffff92e1c69103c0 x1874673756612224/t0(0) o400->MGC192.168.204.160@tcp@192.168.204.160@tcp:26/25 lens 224/224 e 0 to 1 dl 1787832848 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4708.879622] Lustre: 2350:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 82 previous similar messages
[ 4708.895578] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 192.168.204.160@tcp) was lost; in progress operations using this service will fail
[ 4718.060263] Lustre: Evicted from MGS (at 192.168.204.160@tcp) after server handle changed from 0x40546b8e505c5ace to 0x40546b8e505f12ec
[ 4718.078250] Lustre: MGC192.168.204.160@tcp: Connection restored to 192.168.204.160@tcp (at 192.168.204.160@tcp)
[ 4718.089626] Lustre: Skipped 24 previous similar messages
[ 4901.545349] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 08:17:19 (1787833039)
[ 4908.909296] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4926.966309] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 192.168.204.160@tcp) was lost; in progress operations using this service will fail
[ 4927.001243] Lustre: lustre-MDT0000-mdc-ffff92e1c51eb800: Connection to lustre-MDT0000 (at 192.168.204.160@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4927.023515] Lustre: Evicted from MGS (at 192.168.204.160@tcp) after server handle changed from 0x40546b8e505f12ec to 0x40546b8e5063d5e8
[ 4927.038186] Lustre: Skipped 11 previous similar messages
[ 4928.823913] LustreError: lustre-MDT0000-mdc-ffff92e1c51eb800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4950.710130] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 08:18:08 (1787833088)
[ 4951.395814] Lustre: Mounted lustre-client - version 2.17.57_65_g89f8e4a
[ 4959.255821] LustreError: lustre-OST0000-osc-ffff92e1c51eb800: operation ldlm_enqueue to node 192.168.204.160@tcp failed: rc = -107
[ 4959.277556] LustreError: lustre-OST0000-osc-ffff92e1c51eb800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4972.156676] Lustre: Unmounted lustre-client
[ 4978.816161] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 08:18:37 (1787833117)
[ 4985.577320] LustreError: lustre-MDT0000-mdc-ffff92e1c51eb800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4985.594261] LustreError: 60922:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4993.041370] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 08:18:51 (1787833131)
[ 4993.300691] LustreError: 61627:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4997.239165] LustreError: lustre-MDT0000-mdc-ffff92e1c51eb800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4997.260845] LustreError: 61643:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4997.277732] LustreError: 61627:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=1031
[ 4997.296650] LustreError: 61627:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting
[ 4997.328769] LustreError: 61648:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4998.389928] LustreError: 61654:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4998.401367] LustreError: 61654:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5000.624440] LustreError: 61677:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5000.638046] LustreError: 61677:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5005.205076] LustreError: 61721:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5005.218114] LustreError: 61721:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 5014.795635] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 08:19:13 (1787833153)
[ 5055.899955] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5057.583744] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5068.299121] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 08:20:07 (1787833207)
[ 5107.512887] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5109.053796] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5120.000316] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 08:20:58 (1787833258)
[ 5185.791526] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5187.680582] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5196.993309] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 08:22:15 (1787833335)
[ 5239.087804] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5241.191530] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5247.611123] Lustre: Unmounted lustre-client
[ 5310.008120] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5312.751364] Lustre: Mounted lustre-client - version 2.17.57_65_g89f8e4a
[ 5319.351835] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 08:24:18 (1787833458)
[ 5322.024443] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5323.989823] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 08:24:22 (1787833462)
[ 5337.442191] Lustre: lustre-OST0000-osc-ffff92e1c2b9d800: Connection restored to 192.168.204.160@tcp (at 192.168.204.160@tcp)
[ 5337.455402] Lustre: Skipped 12 previous similar messages
[ 5354.566444] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 08:24:52 (1787833492)
[ 5356.165673] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5358.119949] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 08:24:56 (1787833496)
[ 5359.539445] Lustre: *** cfs_fail_loc=805, val=0***
[ 5359.613984] Lustre: Mounted lustre-client - version 2.17.57_65_g89f8e4a
[ 5364.871814] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5385.184233] Lustre: 2348:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787833509/real 1787833509]  req@ffff92e1c9381e00 x1874673758661632/t0(0) o400->lustre-MDT0000-mdc-ffff92e1c2b9d800@192.168.204.160@tcp:12/10 lens 224/224 e 0 to 1 dl 1787833525 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5385.212168] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 192.168.204.160@tcp) was lost; in progress operations using this service will fail
[ 5385.220901] Lustre: 2348:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5385.275174] Lustre: Evicted from MGS (at 192.168.204.160@tcp) after server handle changed from 0x40546b8e5063e2d7 to 0x40546b8e5063e5ee
[ 5394.527967] Lustre: Unmounted lustre-client
[ 5402.200511] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 08:25:40 (1787833540)
[ 5444.095729] LustreError: 2347:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff92e1c56d7c00 x1874673758673280/t94489280516(94489280516) o101->lustre-MDT0000-mdc-ffff92e1c2b9d800@192.168.204.160@tcp:12/10 lens 648/608 e 0 to 0 dl 1787833600 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5451.131715] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5453.409861] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5463.080510] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 08:26:41 (1787833601)
[ 5464.679454] LustreError: lustre-OST0000-osc-ffff92e1c2b9d800: operation ost_write to node 192.168.204.160@tcp failed: rc = -107
[ 5464.698083] LustreError: Skipped 2 previous similar messages
[ 5464.721691] LustreError: lustre-OST0000-osc-ffff92e1c2b9d800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5464.735133] Lustre: 2351:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.160@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5464.799524] LustreError: 72833:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff92e1c2b9d800: namespace resource [0x240000400:0x3a42:0x0].0x0 (ffff92e1c579fa00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5474.005698] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 08:26:52 (1787833612)
[ 5475.611736] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5477.469406] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 08:26:55 (1787833615)
[ 5478.975977] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5480.881565] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 08:26:59 (1787833619)
[ 5482.345203] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5484.659583] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 08:27:02 (1787833622)
[ 5486.376242] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5488.018747] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 08:27:06 (1787833626)
[ 5489.678291] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5491.974784] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 08:27:10 (1787833630)
[ 5493.761431] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5495.901349] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 08:27:14 (1787833634)
[ 5497.315376] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5499.319239] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 08:27:17 (1787833637)
[ 5501.058655] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5503.218865] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 08:27:21 (1787833641)
[ 5505.001513] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5507.016315] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 08:27:25 (1787833645)
[ 5508.862331] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5511.336352] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 08:27:29 (1787833649)
[ 5513.563691] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5516.412102] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 08:27:33 (1787833653)
[ 5519.581744] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5522.118620] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 08:27:40 (1787833660)
[ 5559.814103] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 08:28:18 (1787833698)
[ 5591.330397] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 08:28:49 (1787833729)
[ 5591.954456] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5603.790027] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 08:29:02 (1787833742)
[ 5604.373182] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5615.844789] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 08:29:13 (1787833753)
[ 5616.154752] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5625.476647] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 08:29:24 (1787833764)
[ 5625.850324] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5634.441365] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 08:29:32 (1787833772)
[ 5635.008354] Lustre: *** cfs_fail_loc=510, val=3***
[ 5645.046598] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 08:29:43 (1787833783)
[ 5645.323544] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5655.285581] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 08:29:53 (1787833793)
[ 5655.685073] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5714.401482] Lustre: lustre-OST0001-osc-ffff92e1c2b9d800: Connection to lustre-OST0001 (at 192.168.204.160@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5714.414756] Lustre: Skipped 17 previous similar messages
[ 5721.612748] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 08:31:00 (1787833860)
[ 5722.152057] LustreError: 83044:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5725.081695] LustreError: lustre-MDT0000-mdc-ffff92e1c2b9d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5725.103737] LustreError: 83059:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff92e1c2b9d800: inode [0x20000a042:0xf:0x0] mdc close failed: rc = -108
[ 5725.135042] LustreError: 83059:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff92e1c2b9d800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff92e1c579fe00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5726.239237] LustreError: 83044:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5728.349951] LustreError: 83075:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5731.272816] LustreError: lustre-MDT0000-mdc-ffff92e1c2b9d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5731.292576] LustreError: 83089:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff92e1c2b9d800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff92e1c546a900) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5731.341143] LustreError: 83089:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5732.423156] LustreError: 83075:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5739.355684] LustreError: lustre-MDT0000-mdc-ffff92e1c2b9d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5739.374956] LustreError: 83118:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff92e1c2b9d800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff92e1c6874200) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5739.407352] LustreError: 83118:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5739.835595] LustreError: 83131:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5742.570046] LustreError: lustre-MDT0000-mdc-ffff92e1c2b9d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5742.647481] LustreError: 83150:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5742.656366] LustreError: 83150:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5743.745106] LustreError: 83157:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5743.756238] LustreError: 83157:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5743.903344] LustreError: 83131:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5743.910679] LustreError: 83131:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff92e1c2b9d800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5746.027379] LustreError: 83179:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5746.037350] LustreError: 83179:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5749.636479] LustreError: 83206:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5752.694567] LustreError: lustre-MDT0000-mdc-ffff92e1c2b9d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5752.728555] LustreError: 83220:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff92e1c2b9d800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff92e1c579fc00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5753.703672] LustreError: 83206:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5760.733778] LustreError: lustre-MDT0000-mdc-ffff92e1c2b9d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5761.208528] LustreError: 83263:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5764.368541] LustreError: lustre-MDT0000-mdc-ffff92e1c2b9d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5764.396946] LustreError: 83278:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff92e1c2b9d800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff92e1c579fa00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5764.442664] LustreError: 83278:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5764.469086] LustreError: 83283:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5764.485408] LustreError: 83283:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5765.287141] LustreError: 83263:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5768.539735] LustreError: lustre-MDT0000-mdc-ffff92e1c2b9d800: operation ldlm_enqueue to node 192.168.204.160@tcp failed: rc = -107
[ 5768.550519] LustreError: Skipped 7 previous similar messages
[ 5768.570709] LustreError: lustre-MDT0000-mdc-ffff92e1c2b9d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5769.528723] LustreError: 83291:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff92e1c2b9d800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5777.796188] LustreError: 83362:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5777.807143] LustreError: 83362:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5781.395396] LustreError: lustre-MDT0000-mdc-ffff92e1c2b9d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5781.528275] LustreError: 83385:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5781.540056] LustreError: 83385:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 5781.799161] LustreError: 83362:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5781.811043] LustreError: 83362:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5781.820900] LustreError: 83362:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff92e1c2b9d800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5792.661126] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 08:32:11 (1787833931)
[ 5812.534264] LustreError: 79794:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5812.540479] LustreError: 79794:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5816.599312] LustreError: 79794:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5816.610314] LustreError: 79794:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5823.447897] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 08:32:42 (1787833962)
[ 5873.780904] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 08:33:32 (1787834012)
[ 5937.908488] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 08:34:36 (1787834076)
[ 5972.142566] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 08:35:10 (1787834110)
[ 5972.643250] Lustre: Mounted lustre-client - version 2.17.57_65_g89f8e4a
[ 6096.142868] Lustre: Unmounted lustre-client
[ 6102.958748] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 08:37:21 (1787834241)
[ 6103.501166] LustreError: 87688:0:(osc_request.c:3177:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 6105.997627] LustreError: lustre-OST0000-osc-ffff92e1c2b9d800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6106.014071] LustreError: 87770:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff92e1c2b9d800: namespace resource [0x240000400:0x3a6d:0x0].0x0 (ffff92e1c6805500) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6106.027859] LustreError: 87770:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 6106.437678] LustreError: 87688:0:(osc_request.c:3177:osc_build_rpc()) cfs_fail_timeout interrupted
[ 6106.452967] Lustre: 2351:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.160@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 6113.392778] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 08:37:31 (1787834251)
[ 6171.616034] Lustre: 88354:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787834256/real 1787834256]  req@ffff92e1c57abc00 x1874673758847488/t0(0) o101->lustre-MDT0000-mdc-ffff92e1c2b9d800@192.168.204.160@tcp:12/10 lens 328/344 e 0 to 1 dl 1787834311 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 6171.656933] Lustre: 88354:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 15 previous similar messages
[ 6171.680552] Lustre: lustre-MDT0000-mdc-ffff92e1c2b9d800: Connection restored to 192.168.204.160@tcp (at 192.168.204.160@tcp)
[ 6171.703509] Lustre: Skipped 19 previous similar messages
[ 6179.940570] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 08:38:38 (1787834318)
[ 6181.530472] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 6183.355718] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 08:38:41 (1787834321)
[ 6250.033163] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6252.517160] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 08:39:50 (1787834390)
[ 6312.362405] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 08:40:50 (1787834450)
[ 6315.015451] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6317.748720] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 08:40:55 (1787834455)
[ 6319.821461] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6322.491654] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 08:41:00 (1787834460)
[ 6361.147469] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 08:41:39 (1787834499)
[ 6372.943642] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6381.545897] Lustre: lustre-MDT0000-mdc-ffff92e1c2b9d800: Connection to lustre-MDT0000 (at 192.168.204.160@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6381.563188] Lustre: Skipped 16 previous similar messages
[ 6401.015871] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 192.168.204.160@tcp) was lost; in progress operations using this service will fail
[ 6401.040769] LustreError: Skipped 2 previous similar messages
[ 6401.059489] Lustre: Evicted from MGS (at 192.168.204.160@tcp) after server handle changed from 0x40546b8e5063f12b to 0x40546b8e50640a1b
[ 6401.073450] Lustre: Skipped 2 previous similar messages
[ 6414.125265] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6415.637335] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6425.764942] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 08:42:44 (1787834564)
[ 6427.666118] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6429.757908] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 08:42:48 (1787834568)
[ 6441.977277] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 192.168.204.160@tcp) was lost; in progress operations using this service will fail
[ 6442.013956] Lustre: Evicted from MGS (at 192.168.204.160@tcp) after server handle changed from 0x40546b8e50640a1b to 0x40546b8e50640c75
[ 6458.295215] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 08:43:17 (1787834597)
[ 6507.794900] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 08:44:06 (1787834646)
[ 6515.157775] LustreError: lustre-OST0000-osc-ffff92e1c2b9d800: operation ost_setattr to node 192.168.204.160@tcp failed: rc = -19
[ 6515.170761] LustreError: Skipped 2 previous similar messages
[ 6566.758226] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6569.140716] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6655.999985] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 192.168.204.160@tcp) was lost; in progress operations using this service will fail
[ 6656.015548] LustreError: Skipped 1 previous similar message
[ 6656.047210] Lustre: Evicted from MGS (at 192.168.204.160@tcp) after server handle changed from 0x40546b8e50640fe0 to 0x40546b8e50641e2d
[ 6656.060962] Lustre: Skipped 1 previous similar message
[ 6657.908553] Lustre: 68291:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.160@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6682.286558] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6684.095871] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6731.363895] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6733.568513] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6772.705680] Lustre: 2350:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787834852/real 1787834852]  req@ffff92e1c7092580 x1874673765383424/t0(0) o400->lustre-MDT0000-mdc-ffff92e1c2b9d800@192.168.204.160@tcp:12/10 lens 224/224 e 0 to 1 dl 1787834912 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6772.746733] Lustre: 2350:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 14 previous similar messages
[ 6872.389148] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 08:50:10 (1787835010)
[ 6935.980600] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6940.711790] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 7203.618729] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 08:55:42 (1787835342)
[ 7352.293778] Lustre: lustre-MDT0000-mdc-ffff92e1c2b9d800: Connection to lustre-MDT0000 (at 192.168.204.160@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7352.310614] Lustre: Skipped 6 previous similar messages
[ 7352.321428] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 192.168.204.160@tcp) was lost; in progress operations using this service will fail
[ 7352.333296] LustreError: Skipped 1 previous similar message
[ 7352.344492] Lustre: Evicted from MGS (at 192.168.204.160@tcp) after server handle changed from 0x40546b8e506422da to 0x40546b8e5073c559
[ 7352.360152] Lustre: Skipped 1 previous similar message
[ 7352.370737] Lustre: MGC192.168.204.160@tcp: Connection restored to 192.168.204.160@tcp (at 192.168.204.160@tcp)
[ 7352.382479] Lustre: Skipped 12 previous similar messages
[ 7380.447224] Lustre: 2348:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787835482/real 1787835482]  req@ffff92e1c74c4b40 x1874673776548864/t0(0) o400->lustre-MDT0000-mdc-ffff92e1c2b9d800@192.168.204.160@tcp:12/10 lens 224/224 e 0 to 1 dl 1787835520 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7382.565815] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 08:58:40 (1787835520)
[ 7384.409368] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7386.111807] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 08:58:44 (1787835524)
[ 7388.249582] LustreError: lustre-MDT0000-mdc-ffff92e1c2b9d800: operation ldlm_enqueue to node 192.168.204.160@tcp failed: rc = -107
[ 7388.257549] LustreError: Skipped 160 previous similar messages
[ 7388.272993] LustreError: lustre-MDT0000-mdc-ffff92e1c2b9d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7388.285964] LustreError: 102033:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff92e1c2b9d800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff92e1ee31f100) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 7388.296799] LustreError: 102033:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 7397.470101] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 08:58:55 (1787835535)
[ 7565.293136] LustreError: lustre-OST0000-osc-ffff92e1c2b9d800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7567.170163] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 09:01:45 (1787835705)
[ 7610.324220] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 09:02:29 (1787835749)
[ 7612.165800] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7614.356786] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 09:02:32 (1787835752)
[ 7615.984499] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7617.505593] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 09:02:36 (1787835756)
[ 7656.181539] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 09:03:13 (1787835793)
[ 7700.453472] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 192.168.204.160@tcp) was lost; in progress operations using this service will fail
[ 7700.494679] Lustre: Evicted from MGS (at 192.168.204.160@tcp) after server handle changed from 0x40546b8e5073c559 to 0x40546b8e5073eb0e
[ 7700.508335] Lustre: Skipped 1 previous similar message
[ 7715.730630] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 09:04:14 (1787835854)
[ 7717.273345] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7718.873689] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 09:04:17 (1787835857)
[ 7720.536327] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7722.890271] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 09:04:21 (1787835861)
[ 7733.053943] Lustre: Unmounted lustre-client
[ 7734.967122] Lustre: Mounted lustre-client - version 2.17.57_65_g89f8e4a
[ 7737.962942] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7768.798680] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2