[ 1879.379352] LustreError: lustre-OST0000-osc-ffff98eb0ab27000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1879.397936] Lustre: 2351:0:(llite_lib.c:4327:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.140@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1886.960552] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 12:36:42 (1784047002)
[ 1889.707416] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1891.543481] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 12:36:47 (1784047007)
[ 1893.275935] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1895.078990] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 12:36:50 (1784047010)
[ 1918.946819] LustreError: MGC192.168.203.140@tcp: Connection to MGS (at 192.168.203.140@tcp) was lost; in progress operations using this service will fail
[ 1918.975403] Lustre: Evicted from MGS (at 192.168.203.140@tcp) after server handle changed from 0xd4ffa79a9ad9896 to 0xd4ffa79a9adaff0
[ 1920.939452] Lustre: 15888:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.140@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2028.221110] LustreError: lustre-MDT0000-mdc-ffff98eb0ab27000: operation ldlm_enqueue to node 192.168.203.140@tcp failed: rc = -19
[ 2028.228196] LustreError: Skipped 3 previous similar messages
[ 2045.920267] LustreError: MGC192.168.203.140@tcp: Connection to MGS (at 192.168.203.140@tcp) was lost; in progress operations using this service will fail
[ 2056.105246] Lustre: Evicted from MGS (at 192.168.203.140@tcp) after server handle changed from 0xd4ffa79a9adaff0 to 0xd4ffa79a9afa725
[ 2072.032985] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 12:39:47 (1784047187)
[ 2072.603937] Lustre: *** cfs_fail_loc=305, val=0***
[ 2072.606421] Lustre: Skipped 2 previous similar messages
[ 2092.328812] LustreError: lustre-OST0001-osc-ffff98eb0ab27000: operation ost_connect to node 192.168.203.140@tcp failed: rc = -75
[ 2118.112401] LustreError: MGC192.168.203.140@tcp: Connection to MGS (at 192.168.203.140@tcp) was lost; in progress operations using this service will fail
[ 2118.148317] Lustre: Evicted from MGS (at 192.168.203.140@tcp) after server handle changed from 0xd4ffa79a9afa725 to 0xd4ffa79a9afac96
[ 2130.642159] Lustre: DEBUG MARKER: oleg340-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2132.746057] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2141.147338] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 12:40:56 (1784047256)
[ 2153.966572] LustreError: MGC192.168.203.140@tcp: Connection to MGS (at 192.168.203.140@tcp) was lost; in progress operations using this service will fail
[ 2153.988224] Lustre: lustre-MDT0000-mdc-ffff98eb0ab27000: Connection to lustre-MDT0000 (at 192.168.203.140@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2153.997275] Lustre: Evicted from MGS (at 192.168.203.140@tcp) after server handle changed from 0xd4ffa79a9afac96 to 0xd4ffa79a9afb254
[ 2154.002130] Lustre: Skipped 12 previous similar messages
[ 2154.025294] Lustre: MGC192.168.203.140@tcp: Connection restored to 192.168.203.140@tcp (at 192.168.203.140@tcp)
[ 2154.042710] Lustre: Skipped 15 previous similar messages
[ 2154.087463] LustreError: lustre-MDT0000-mdc-ffff98eb0ab27000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2175.502327] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 12:41:31 (1784047291)
[ 2189.815032] LustreError: lustre-OST0000-osc-ffff98eb0ab27000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2204.161719] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 12:42:00 (1784047320)
[ 2235.872612] LustreError: MGC192.168.203.140@tcp: Connection to MGS (at 192.168.203.140@tcp) was lost; in progress operations using this service will fail
[ 2246.140037] Lustre: Evicted from MGS (at 192.168.203.140@tcp) after server handle changed from 0xd4ffa79a9afb254 to 0xd4ffa79a9b007ab
[ 2266.046564] Lustre: DEBUG MARKER: oleg340-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2268.081178] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2333.812268] LustreError: lustre-MDT0000-mdc-ffff98eb0ab27000: operation ldlm_enqueue to node 192.168.203.140@tcp failed: rc = -19
[ 2333.826697] LustreError: Skipped 1 previous similar message
[ 2353.640640] LustreError: MGC192.168.203.140@tcp: Connection to MGS (at 192.168.203.140@tcp) was lost; in progress operations using this service will fail
[ 2353.672922] Lustre: Evicted from MGS (at 192.168.203.140@tcp) after server handle changed from 0xd4ffa79a9b007ab to 0xd4ffa79a9b1dddf
[ 2372.835760] Lustre: DEBUG MARKER: oleg340-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2375.261910] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2460.136728] LustreError: MGC192.168.203.140@tcp: Connection to MGS (at 192.168.203.140@tcp) was lost; in progress operations using this service will fail
[ 2460.154723] Lustre: Evicted from MGS (at 192.168.203.140@tcp) after server handle changed from 0xd4ffa79a9b1dddf to 0xd4ffa79a9b3d3cb
[ 2462.136407] Lustre: 15888:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.140@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2487.777789] Lustre: DEBUG MARKER: oleg340-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2489.661054] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2518.560988] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 12:47:14 (1784047634)
[ 2542.048331] Lustre: 2351:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784047643/real 1784047643]  req@ffff98eb0bd74380 x1870706954608128/t0(0) o400->MGC192.168.203.140@tcp@192.168.203.140@tcp:26/25 lens 224/224 e 0 to 1 dl 1784047659 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2542.105261] Lustre: 2351:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 12 previous similar messages
[ 2542.113510] LustreError: MGC192.168.203.140@tcp: Connection to MGS (at 192.168.203.140@tcp) was lost; in progress operations using this service will fail
[ 2542.132306] Lustre: Evicted from MGS (at 192.168.203.140@tcp) after server handle changed from 0xd4ffa79a9b3d3cb to 0xd4ffa79a9b4afd4
[ 2544.122372] Lustre: 15888:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.140@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2559.217396] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2589.152441] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2596.586262] LustreError: lustre-MDT0000-mdc-ffff98eb0ab27000: operation ldlm_enqueue to node 192.168.203.140@tcp failed: rc = -19
[ 2596.596590] LustreError: Skipped 6 previous similar messages
[ 2628.925384] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2673.278551] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2713.056625] LustreError: MGC192.168.203.140@tcp: Connection to MGS (at 192.168.203.140@tcp) was lost; in progress operations using this service will fail
[ 2713.065935] LustreError: Skipped 3 previous similar messages
[ 2723.273238] Lustre: Evicted from MGS (at 192.168.203.140@tcp) after server handle changed from 0xd4ffa79a9b5923c to 0xd4ffa79a9b6566d
[ 2723.288367] Lustre: Skipped 3 previous similar messages
[ 2724.423700] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2773.710840] Lustre: lustre-MDT0000-mdc-ffff98eb0ab27000: Connection restored to 192.168.203.140@tcp (at 192.168.203.140@tcp)
[ 2773.722964] Lustre: Skipped 18 previous similar messages
[ 2782.262100] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2815.808557] Lustre: lustre-MDT0000-mdc-ffff98eb0ab27000: Connection to lustre-MDT0000 (at 192.168.203.140@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2815.826996] Lustre: Skipped 10 previous similar messages
[ 2838.078866] Lustre: 15888:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.140@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2881.654617] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 12:53:16 (1784047996)
[ 2943.881393] Lustre: DEBUG MARKER: oleg340-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 2946.294431] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3227.498227] LustreError: lustre-OST0000-osc-ffff98eb0ab27000: operation ldlm_enqueue to node 192.168.203.140@tcp failed: rc = -19
[ 3227.516885] LustreError: Skipped 7 previous similar messages
[ 3265.890080] Lustre: DEBUG MARKER: oleg340-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3268.411240] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3560.209617] Lustre: lustre-OST0000-osc-ffff98eb0ab27000: Connection to lustre-OST0000 (at 192.168.203.140@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3560.233045] Lustre: Skipped 2 previous similar messages
[ 3583.159522] Lustre: lustre-OST0000-osc-ffff98eb0ab27000: Connection restored to 192.168.203.140@tcp (at 192.168.203.140@tcp)
[ 3583.171673] Lustre: Skipped 5 previous similar messages
[ 3600.393233] Lustre: DEBUG MARKER: oleg340-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3602.774625] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3855.951624] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 13:09:31 (1784048971)
[ 3873.760576] Lustre: 50393:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784048974/real 1784048974]  req@ffff98eb0bc8e680 x1870706974366592/t0(0) o101->lustre-MDT0000-mdc-ffff98eb0ab27000@192.168.203.140@tcp:12/10 lens 576/1152 e 0 to 1 dl 1784048990 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3873.811026] Lustre: 50393:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 3881.835536] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 13:09:57 (1784048997)
[ 3909.964986] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 13:10:25 (1784049025)
[ 3934.529323] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 13:10:50 (1784049050)
[ 3935.024036] Lustre: Mounted lustre-client
[ 3965.920174] Lustre: 2348:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784049067/real 1784049067]  req@ffff98eb0bc8d500 x1870706974386688/t0(0) o400->lustre-MDT0000-mdc-ffff98eb0ab27000@192.168.203.140@tcp:12/10 lens 224/224 e 0 to 1 dl 1784049083 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3965.922139] LustreError: MGC192.168.203.140@tcp: Connection to MGS (at 192.168.203.140@tcp) was lost; in progress operations using this service will fail
[ 3965.973727] Lustre: 2348:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 3966.001800] LustreError: Skipped 2 previous similar messages
[ 3976.185226] Lustre: Evicted from MGS (at 192.168.203.140@tcp) after server handle changed from 0xd4ffa79a9b7bca1 to 0xd4ffa79a9cbcc71
[ 3976.214769] Lustre: Skipped 2 previous similar messages
[ 3981.432464] Lustre: DEBUG MARKER: oleg340-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 3982.964274] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3984.922601] Lustre: Unmounted lustre-client
[ 3990.792922] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 13:11:46 (1784049106)
[ 4126.689202] Lustre: 2351:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784049227/real 1784049227]  req@ffff98eb0bd51f80 x1870706974418944/t0(0) o4->lustre-OST0000-osc-ffff98eb0ab27000@192.168.203.140@tcp:6/4 lens 488/448 e 0 to 1 dl 1784049243 ref 2 fl Rpc:XQr/602/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4126.754988] Lustre: 2351:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 63 previous similar messages
[ 4174.816214] Lustre: lustre-OST0000-osc-ffff98eb0ab27000: Connection to lustre-OST0000 (at 192.168.203.140@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4174.841378] Lustre: Skipped 15 previous similar messages
[ 4191.225285] Lustre: lustre-OST0000-osc-ffff98eb0ab27000: Connection restored to 192.168.203.140@tcp (at 192.168.203.140@tcp)
[ 4191.242732] Lustre: Skipped 17 previous similar messages
[ 4281.147371] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 13:16:36 (1784049396)
[ 4334.172622] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 13:17:29 (1784049449)
[ 4338.095849] Lustre: Unmounted lustre-client
[ 4361.867704] Lustre: Mounted lustre-client
[ 4367.764668] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 13:18:03 (1784049483)
[ 4368.144633] LustreError: 56294:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4369.203042] LustreError: 56294:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4369.421197] Lustre: *** cfs_fail_loc=305, val=0***
[ 4395.179720] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 13:18:30 (1784049510)
[ 4395.712063] Lustre: Mounted lustre-client
[ 4397.906759] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4408.207455] Lustre: Unmounted lustre-client
[ 4414.913457] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 13:18:50 (1784049530)
[ 4501.442503] LustreError: lustre-MDT0000-mdc-ffff98eb0bdfe000: operation mds_reint to node 192.168.203.140@tcp failed: rc = -19
[ 4501.455637] LustreError: Skipped 1 previous similar message
[ 4519.843278] Lustre: 2349:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784049620/real 1784049620]  req@ffff98eb0caba300 x1870706977094144/t0(0) o400->MGC192.168.203.140@tcp@192.168.203.140@tcp:26/25 lens 224/224 e 0 to 1 dl 1784049636 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4519.868585] Lustre: 2349:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 84 previous similar messages
[ 4519.874257] LustreError: MGC192.168.203.140@tcp: Connection to MGS (at 192.168.203.140@tcp) was lost; in progress operations using this service will fail
[ 4530.098712] Lustre: Evicted from MGS (at 192.168.203.140@tcp) after server handle changed from 0xd4ffa79a9cbe610 to 0xd4ffa79a9ce98f5
[ 4697.797846] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 13:23:33 (1784049813)
[ 4702.206634] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4717.551693] LustreError: MGC192.168.203.140@tcp: Connection to MGS (at 192.168.203.140@tcp) was lost; in progress operations using this service will fail
[ 4717.583081] Lustre: Evicted from MGS (at 192.168.203.140@tcp) after server handle changed from 0xd4ffa79a9ce98f5 to 0xd4ffa79a9d365de
[ 4717.630440] LustreError: lustre-MDT0000-mdc-ffff98eb0bdfe000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4729.570380] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 13:24:05 (1784049845)
[ 4729.961343] Lustre: Mounted lustre-client
[ 4737.210389] LustreError: lustre-OST0000-osc-ffff98eb0bdfe000: operation ldlm_enqueue to node 192.168.203.140@tcp failed: rc = -107
[ 4737.240018] LustreError: lustre-OST0000-osc-ffff98eb0bdfe000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4750.919236] Lustre: Unmounted lustre-client
[ 4760.695455] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 13:24:36 (1784049876)
[ 4768.215801] LustreError: lustre-MDT0000-mdc-ffff98eb0bdfe000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4768.234045] LustreError: 60667:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4776.387775] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 13:24:51 (1784049891)
[ 4776.721202] Lustre: lustre-MDT0000-mdc-ffff98eb0bdfe000: Connection to lustre-MDT0000 (at 192.168.203.140@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4776.747490] Lustre: Skipped 15 previous similar messages
[ 4776.755171] LustreError: 61373:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4782.048129] LustreError: 61373:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=0
[ 4782.054582] LustreError: 61373:0:(import.c:711:ptlrpc_connect_import_locked()) already connected
[ 4782.639057] LustreError: lustre-MDT0000-mdc-ffff98eb0bdfe000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4782.682772] LustreError: 61390:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4782.793955] LustreError: 61395:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4783.960684] LustreError: 61401:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4783.980222] LustreError: 61401:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4786.393126] LustreError: 61424:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4786.401051] LustreError: 61424:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4791.201680] LustreError: 61469:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4791.212539] LustreError: 61469:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4792.811145] Lustre: lustre-MDT0000-mdc-ffff98eb0bdfe000: Connection restored to 192.168.203.140@tcp (at 192.168.203.140@tcp)
[ 4792.825340] Lustre: Skipped 16 previous similar messages
[ 4801.962262] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 13:25:17 (1784049917)
[ 4845.673532] Lustre: DEBUG MARKER: oleg340-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4849.894891] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4862.399590] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 13:26:17 (1784049977)
[ 4900.272730] Lustre: DEBUG MARKER: oleg340-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4901.724149] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4913.476636] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 13:27:09 (1784050029)
[ 4982.966663] Lustre: DEBUG MARKER: oleg340-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4985.242436] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4994.110861] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 13:28:29 (1784050109)
[ 5031.121650] Lustre: DEBUG MARKER: oleg340-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5033.004769] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5039.832834] Lustre: Unmounted lustre-client
[ 5095.390904] Lustre: DEBUG MARKER: oleg340-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5098.057728] Lustre: Mounted lustre-client
[ 5104.421462] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 13:30:20 (1784050220)
[ 5106.754450] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5108.837808] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 13:30:24 (1784050224)
[ 5141.398840] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 13:30:57 (1784050257)
[ 5142.687853] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5144.302324] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 13:31:00 (1784050260)
[ 5145.944645] Lustre: *** cfs_fail_loc=805, val=0***
[ 5145.999720] Lustre: Mounted lustre-client
[ 5151.791439] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5172.704251] Lustre: 2349:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784050273/real 1784050273]  req@ffff98eb04c84a80 x1870706979150336/t0(0) o400->lustre-MDT0000-mdc-ffff98eb0b540000@192.168.203.140@tcp:12/10 lens 224/224 e 0 to 1 dl 1784050289 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5172.708773] LustreError: MGC192.168.203.140@tcp: Connection to MGS (at 192.168.203.140@tcp) was lost; in progress operations using this service will fail
[ 5172.752079] Lustre: 2349:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 5172.800640] Lustre: Evicted from MGS (at 192.168.203.140@tcp) after server handle changed from 0xd4ffa79a9d36fcb to 0xd4ffa79a9d371e6
[ 5174.514274] Lustre: 67990:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.140@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 5190.485559] Lustre: Unmounted lustre-client
[ 5198.643872] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 13:31:54 (1784050314)
[ 5225.007908] LustreError: 2347:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff98eb04f35180 x1870706979163008/t94489280516(94489280516) o101->lustre-MDT0000-mdc-ffff98eb0b540000@192.168.203.140@tcp:12/10 lens 664/608 e 0 to 0 dl 1784050358 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5238.233883] Lustre: DEBUG MARKER: oleg340-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5240.104338] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5248.109558] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 13:32:44 (1784050364)
[ 5249.592085] LustreError: lustre-OST0000-osc-ffff98eb0b540000: operation ost_write to node 192.168.203.140@tcp failed: rc = -107
[ 5249.598988] LustreError: Skipped 2 previous similar messages
[ 5249.611359] LustreError: lustre-OST0000-osc-ffff98eb0b540000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5249.629978] Lustre: 2350:0:(llite_lib.c:4327:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.140@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5249.658340] LustreError: 72503:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff98eb0b540000: namespace resource [0x240000400:0x3f82:0x0].0x0 (ffff98eb058bf400) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5257.067092] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 13:32:52 (1784050372)
[ 5258.319562] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5260.235426] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 13:32:55 (1784050375)
[ 5261.927227] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5263.562596] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 13:32:59 (1784050379)
[ 5264.958646] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5266.706808] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 13:33:02 (1784050382)
[ 5268.410760] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5270.547834] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 13:33:06 (1784050386)
[ 5271.943582] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5273.431557] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 13:33:09 (1784050389)
[ 5275.074956] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5277.471495] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 13:33:12 (1784050392)
[ 5279.791108] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5281.685810] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 13:33:17 (1784050397)
[ 5283.488353] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5285.818797] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 13:33:21 (1784050401)
[ 5287.902325] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5289.987594] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 13:33:25 (1784050405)
[ 5291.558754] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5293.510812] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 13:33:29 (1784050409)
[ 5294.878277] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5296.916266] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 13:33:32 (1784050412)
[ 5299.453292] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5301.115812] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 13:33:36 (1784050416)
[ 5322.237953] Lustre: 67990:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.140@tcp: IR log lustre-cliir failed, not fatal: rc = -5
[ 5332.387851] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 13:34:07 (1784050447)
[ 5364.381531] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 13:34:40 (1784050480)
[ 5364.912289] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5375.414303] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 13:34:51 (1784050491)
[ 5375.803553] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5385.609233] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 13:35:01 (1784050501)
[ 5385.849366] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5393.856414] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 13:35:09 (1784050509)
[ 5394.471433] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5403.545502] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 13:35:19 (1784050519)
[ 5404.147107] Lustre: *** cfs_fail_loc=510, val=3***
[ 5413.303801] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 13:35:28 (1784050528)
[ 5413.659693] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5472.224250] Lustre: lustre-OST0001-osc-ffff98eb0b540000: Connection to lustre-OST0001 (at 192.168.203.140@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5472.239546] Lustre: Skipped 10 previous similar messages
[ 5480.051584] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 13:36:35 (1784050595)
[ 5480.569320] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5548.150912] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 13:37:43 (1784050663)
[ 5548.845067] LustreError: 82702:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5551.844310] LustreError: lustre-MDT0000-mdc-ffff98eb0b540000: operation ldlm_enqueue to node 192.168.203.140@tcp failed: rc = -107
[ 5551.869846] LustreError: lustre-MDT0000-mdc-ffff98eb0b540000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5551.888503] LustreError: 82716:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff98eb0b540000: inode [0x20000a042:0x8:0x0] mdc close failed: rc = -108
[ 5551.924792] LustreError: 82716:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff98eb0b540000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff98eb05a6d500) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5551.956366] LustreError: 82721:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5551.964471] Lustre: lustre-MDT0000-mdc-ffff98eb0b540000: Connection restored to 192.168.203.140@tcp (at 192.168.203.140@tcp)
[ 5551.973260] LustreError: 82721:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5552.001604] Lustre: Skipped 11 previous similar messages
[ 5552.928133] LustreError: 82702:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5555.056220] LustreError: 82732:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5558.220957] LustreError: lustre-MDT0000-mdc-ffff98eb0b540000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5558.240784] LustreError: 82746:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff98eb0b540000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff98eb05a6d900) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5558.267945] LustreError: 82746:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5559.123876] LustreError: 82732:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5566.204762] LustreError: lustre-MDT0000-mdc-ffff98eb0b540000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5566.764780] LustreError: 82788:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5569.842866] LustreError: lustre-MDT0000-mdc-ffff98eb0b540000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5569.958949] LustreError: 82807:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5570.768163] LustreError: 82788:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5570.779276] LustreError: 82788:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff98eb0b540000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5570.804973] LustreError: 82788:0:(file.c:251:ll_close_inode_openhandle()) Skipped 1 previous similar message
[ 5572.184135] LustreError: 82824:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5572.208915] LustreError: 82824:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 12 previous similar messages
[ 5577.348560] LustreError: 82862:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5580.456309] LustreError: lustre-MDT0000-mdc-ffff98eb0b540000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5580.471082] LustreError: 82876:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff98eb0b540000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff98eb05a6de00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5580.529514] LustreError: 82876:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5581.360119] LustreError: 82862:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5588.675504] LustreError: lustre-MDT0000-mdc-ffff98eb0b540000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5588.692530] LustreError: 82905:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff98eb0b540000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff98eb058bfd00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5588.720661] LustreError: 82905:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5589.219515] LustreError: 82918:0:(ldlm_flock.c:858:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5592.559142] LustreError: lustre-MDT0000-mdc-ffff98eb0b540000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5593.312155] LustreError: 82918:0:(ldlm_flock.c:858:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5596.589376] LustreError: lustre-MDT0000-mdc-ffff98eb0b540000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5596.688983] LustreError: 82965:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5596.695335] LustreError: 82965:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5597.529789] LustreError: 82945:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff98eb0b540000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5605.578491] LustreError: 83016:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5605.591848] LustreError: 83016:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5609.522904] LustreError: lustre-MDT0000-mdc-ffff98eb0b540000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5609.628827] LustreError: 83039:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5609.645885] LustreError: 83039:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5609.680088] LustreError: 83016:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5609.696893] LustreError: 83016:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5609.733196] LustreError: 83016:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff98eb0b540000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5622.334736] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 13:38:57 (1784050737)
[ 5643.141743] LustreError: 67987:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5643.156293] LustreError: 67987:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5647.240751] LustreError: 67987:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5647.253281] LustreError: 67987:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5654.522665] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 13:39:30 (1784050770)
[ 5708.330585] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 13:40:23 (1784050823)
[ 5774.568775] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 13:41:29 (1784050889)
[ 5812.863896] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 13:42:07 (1784050927)
[ 5813.442656] Lustre: Mounted lustre-client
[ 5937.293479] Lustre: Unmounted lustre-client
[ 5945.183514] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 13:44:20 (1784051060)
[ 5945.697082] LustreError: 87350:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5948.278206] LustreError: lustre-OST0000-osc-ffff98eb0b540000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5948.287738] LustreError: 87438:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff98eb0b540000: namespace resource [0x240000400:0x3fad:0x0].0x0 (ffff98eb058bfb00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5948.297723] LustreError: 87438:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 2 previous similar messages
[ 5948.536325] LustreError: 87350:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5948.544401] Lustre: 2350:0:(llite_lib.c:4327:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.140@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5955.275592] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 13:44:30 (1784051070)
[ 6012.896160] Lustre: 88017:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784051074/real 1784051074]  req@ffff98eb04f36680 x1870706979339008/t0(0) o101->lustre-MDT0000-mdc-ffff98eb0b540000@192.168.203.140@tcp:12/10 lens 328/344 e 0 to 1 dl 1784051130 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 6012.920474] Lustre: 88017:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 18 previous similar messages
[ 6021.840760] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 13:45:37 (1784051137)
[ 6023.490496] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 6025.733221] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 13:45:41 (1784051141)
[ 6083.552555] Lustre: lustre-MDT0000-mdc-ffff98eb0b540000: Connection to lustre-MDT0000 (at 192.168.203.140@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6083.561830] Lustre: Skipped 15 previous similar messages
[ 6090.892343] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6092.777709] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 13:46:48 (1784051208)
[ 6151.720738] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 13:47:47 (1784051267)
[ 6154.081462] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6155.985539] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 13:47:51 (1784051271)
[ 6157.501487] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6159.483032] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 13:47:54 (1784051274)
[ 6197.253178] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 13:48:32 (1784051312)
[ 6212.900791] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6237.152731] LustreError: MGC192.168.203.140@tcp: Connection to MGS (at 192.168.203.140@tcp) was lost; in progress operations using this service will fail
[ 6237.170671] LustreError: Skipped 3 previous similar messages
[ 6242.376029] Lustre: lustre-MDT0000-mdc-ffff98eb0b540000: Connection restored to 192.168.203.140@tcp (at 192.168.203.140@tcp)
[ 6242.387381] Lustre: Skipped 15 previous similar messages
[ 6247.464476] Lustre: Evicted from MGS (at 192.168.203.140@tcp) after server handle changed from 0xd4ffa79a9d37e42 to 0xd4ffa79a9d3990e
[ 6247.471211] Lustre: Skipped 3 previous similar messages
[ 6254.323228] Lustre: DEBUG MARKER: oleg340-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6255.905474] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6265.291342] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 13:49:41 (1784051381)
[ 6267.620147] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6269.234829] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 13:49:45 (1784051385)
[ 6283.267830] LustreError: MGC192.168.203.140@tcp: Connection to MGS (at 192.168.203.140@tcp) was lost; in progress operations using this service will fail
[ 6283.288456] Lustre: Evicted from MGS (at 192.168.203.140@tcp) after server handle changed from 0xd4ffa79a9d3990e to 0xd4ffa79a9d39b29
[ 6293.888600] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 13:50:09 (1784051409)
[ 6333.346158] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 13:50:49 (1784051449)
[ 6340.893366] LustreError: lustre-OST0000-osc-ffff98eb0b540000: operation ost_setattr to node 192.168.203.140@tcp failed: rc = -107
[ 6340.910131] LustreError: Skipped 9 previous similar messages
[ 6391.060479] Lustre: DEBUG MARKER: oleg340-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6394.140838] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6478.688237] LustreError: MGC192.168.203.140@tcp: Connection to MGS (at 192.168.203.140@tcp) was lost; in progress operations using this service will fail
[ 6478.701023] LustreError: Skipped 1 previous similar message
[ 6488.380142] Lustre: Evicted from MGS (at 192.168.203.140@tcp) after server handle changed from 0xd4ffa79a9d3a085 to 0xd4ffa79a9d3af50
[ 6488.395269] Lustre: Skipped 1 previous similar message
[ 6491.923809] Lustre: DEBUG MARKER: oleg340-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6493.143984] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6515.706599] Lustre: 67990:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.140@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6536.025523] Lustre: DEBUG MARKER: oleg340-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6537.391080] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6656.416149] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 13:56:12 (1784051772)
[ 6714.873602] Lustre: DEBUG MARKER: oleg340-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6718.454487] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6966.991754] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 14:01:22 (1784052082)
[ 7092.723404] Lustre: lustre-MDT0000-mdc-ffff98eb0b540000: Connection to lustre-MDT0000 (at 192.168.203.140@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7092.738674] Lustre: Skipped 8 previous similar messages
[ 7102.951856] LustreError: MGC192.168.203.140@tcp: Connection to MGS (at 192.168.203.140@tcp) was lost; in progress operations using this service will fail
[ 7102.963667] LustreError: Skipped 1 previous similar message
[ 7102.973741] Lustre: Evicted from MGS (at 192.168.203.140@tcp) after server handle changed from 0xd4ffa79a9d3af8f to 0xd4ffa79a9e3536c
[ 7102.983554] Lustre: Skipped 1 previous similar message
[ 7102.993057] Lustre: MGC192.168.203.140@tcp: Connection restored to 192.168.203.140@tcp (at 192.168.203.140@tcp)
[ 7102.999971] Lustre: Skipped 9 previous similar messages
[ 7141.694716] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 14:04:17 (1784052257)
[ 7142.932739] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7144.493898] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 14:04:20 (1784052260)
[ 7146.257939] LustreError: lustre-MDT0000-mdc-ffff98eb0b540000: operation ldlm_enqueue to node 192.168.203.140@tcp failed: rc = -107
[ 7146.270345] LustreError: Skipped 119 previous similar messages
[ 7146.286907] LustreError: lustre-MDT0000-mdc-ffff98eb0b540000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7154.145651] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 14:04:29 (1784052269)
[ 7321.454732] LustreError: lustre-OST0000-osc-ffff98eb0b540000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7322.766550] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 14:07:18 (1784052438)
[ 7347.170802] Lustre: 2349:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784052444/real 1784052444]  req@ffff98eb1e4f7b80 x1870706997046784/t0(0) o4->lustre-OST0000-osc-ffff98eb0b540000@192.168.203.140@tcp:6/4 lens 4584/448 e 0 to 1 dl 1784052464 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7347.205467] Lustre: 2349:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 26 previous similar messages
[ 7364.752193] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 14:08:00 (1784052480)
[ 7365.990747] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7367.255380] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 14:08:03 (1784052483)
[ 7368.346485] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7369.715172] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 14:08:05 (1784052485)
[ 7399.964634] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 14:08:36 (1784052516)
[ 7439.339579] LustreError: MGC192.168.203.140@tcp: Connection to MGS (at 192.168.203.140@tcp) was lost; in progress operations using this service will fail
[ 7439.381454] Lustre: Evicted from MGS (at 192.168.203.140@tcp) after server handle changed from 0xd4ffa79a9e3536c to 0xd4ffa79a9e37b19
[ 7439.401164] Lustre: Skipped 1 previous similar message
[ 7452.946395] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 14:09:28 (1784052568)
[ 7454.455235] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7456.215701] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 14:09:31 (1784052571)
[ 7457.734822] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7459.633688] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 14:09:35 (1784052575)
[ 7471.803312] Lustre: Unmounted lustre-client
[ 7473.758596] Lustre: Mounted lustre-client
[ 7476.606162] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7494.112217] Lustre: 2351:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784052595/real 1784052595]  req@ffff98eb1890f100 x1870706997232896/t0(0) o400->MGC192.168.203.140@tcp@192.168.203.140@tcp:26/25 lens 224/224 e 0 to 1 dl 1784052611 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7504.775516] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2