[ 1935.199461] LustreError: lustre-OST0000-osc-ffff8a8f08cae800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1936.922881] Lustre: DEBUG MARKER: recovery-small test_24b: @@@@@@ IGNORE (bz5494): multiop didn't fail fsync: 5 or close: 0
[ 1940.772602] Lustre: DEBUG MARKER: recovery-small test_24b: @@@@@@ FAIL: no discarded dirty page found!
[ 1953.963265] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 15:14:08 (1789499648)
[ 1956.264407] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1957.899948] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 15:14:12 (1789499652)
[ 1959.372415] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1961.405701] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 15:14:16 (1789499656)
[ 1964.722265] LustreError: lustre-MDT0000-mdc-ffff8a8f08cae800: operation mds_reint to node 192.168.204.157@tcp failed: rc = -19
[ 1964.733074] LustreError: Skipped 1 previous similar message
[ 1983.969571] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 192.168.204.157@tcp) was lost; in progress operations using this service will fail
[ 1994.150111] Lustre: Evicted from MGS (at 192.168.204.157@tcp) after server handle changed from 0x3424367dca8f4d05 to 0x3424367dca8f662d
[ 2099.595332] LustreError: lustre-MDT0000-mdc-ffff8a8f08cae800: operation ldlm_enqueue to node 192.168.204.157@tcp failed: rc = -19
[ 2099.605733] LustreError: Skipped 1 previous similar message
[ 2116.063268] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 192.168.204.157@tcp) was lost; in progress operations using this service will fail
[ 2126.262604] Lustre: Evicted from MGS (at 192.168.204.157@tcp) after server handle changed from 0x3424367dca8f662d to 0x3424367dca9181ab
[ 2143.057672] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 15:17:17 (1789499837)
[ 2143.401505] Lustre: *** cfs_fail_loc=305, val=0***
[ 2143.405418] Lustre: Skipped 2 previous similar messages
[ 2186.744621] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 192.168.204.157@tcp) was lost; in progress operations using this service will fail
[ 2186.761712] Lustre: lustre-MDT0000-mdc-ffff8a8f08cae800: Connection to lustre-MDT0000 (at 192.168.204.157@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2186.767762] Lustre: Evicted from MGS (at 192.168.204.157@tcp) after server handle changed from 0x3424367dca9181ab to 0x3424367dca91879a
[ 2186.773335] Lustre: Skipped 11 previous similar messages
[ 2186.808530] Lustre: MGC192.168.204.157@tcp: Connection restored to 192.168.204.157@tcp (at 192.168.204.157@tcp)
[ 2186.815891] Lustre: Skipped 13 previous similar messages
[ 2200.950036] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2202.382603] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2210.521598] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 15:18:25 (1789499905)
[ 2227.692464] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 192.168.204.157@tcp) was lost; in progress operations using this service will fail
[ 2227.730416] Lustre: Evicted from MGS (at 192.168.204.157@tcp) after server handle changed from 0x3424367dca91879a to 0x3424367dca918ca2
[ 2227.783955] LustreError: lustre-MDT0000-mdc-ffff8a8f08cae800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2302.310061] Lustre: DEBUG MARKER: recovery-small test_29a: @@@@@@ FAIL: mds1: import is not in FULL state after 50
[ 2319.252763] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 15:20:13 (1789500013)
[ 2354.838183] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 15:20:49 (1789500049)
[ 2368.004543] LustreError: lustre-MDT0000-mdc-ffff8a8f08cae800: operation mds_reint to node 192.168.204.157@tcp failed: rc = -19
[ 2368.019654] LustreError: Skipped 2 previous similar messages
[ 2387.425180] Lustre: 2365:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789500067/real 1789500067]  req@ffff8a8f08188000 x1876424432979328/t0(0) o400->MGC192.168.204.157@tcp@192.168.204.157@tcp:26/25 lens 224/224 e 0 to 1 dl 1789500083 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2387.458607] Lustre: 2365:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 13 previous similar messages
[ 2387.474118] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 192.168.204.157@tcp) was lost; in progress operations using this service will fail
[ 2387.501990] Lustre: Evicted from MGS (at 192.168.204.157@tcp) after server handle changed from 0x3424367dca918ca2 to 0x3424367dca91dc7a
[ 2388.724385] Lustre: 15950:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.157@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2413.244522] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2415.607059] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2498.015254] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 192.168.204.157@tcp) was lost; in progress operations using this service will fail
[ 2508.229180] Lustre: Evicted from MGS (at 192.168.204.157@tcp) after server handle changed from 0x3424367dca91dc7a to 0x3424367dca939312
[ 2525.480458] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2527.697561] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2610.208180] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 192.168.204.157@tcp) was lost; in progress operations using this service will fail
[ 2620.404409] Lustre: Evicted from MGS (at 192.168.204.157@tcp) after server handle changed from 0x3424367dca939312 to 0x3424367dca9544d3
[ 2640.162328] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2642.357196] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2672.804812] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 15:26:07 (1789500367)
[ 2677.498577] LustreError: lustre-MDT0000-mdc-ffff8a8f08cae800: operation ldlm_enqueue to node 192.168.204.157@tcp failed: rc = -19
[ 2677.506265] LustreError: Skipped 6 previous similar messages
[ 2695.146812] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 192.168.204.157@tcp) was lost; in progress operations using this service will fail
[ 2695.163035] Lustre: Evicted from MGS (at 192.168.204.157@tcp) after server handle changed from 0x3424367dca9544d3 to 0x3424367dca96169b
[ 2705.211853] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2739.350841] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2767.177697] Lustre: 15950:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.157@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2784.904872] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2798.248956] Lustre: lustre-MDT0000-mdc-ffff8a8f08cae800: Connection to lustre-MDT0000 (at 192.168.204.157@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2798.266363] Lustre: Skipped 6 previous similar messages
[ 2826.167068] Lustre: Evicted from MGS (at 192.168.204.157@tcp) after server handle changed from 0x3424367dca96594c to 0x3424367dca96e0a6
[ 2826.182898] Lustre: Skipped 2 previous similar messages
[ 2826.187755] Lustre: MGC192.168.204.157@tcp: Connection restored to 192.168.204.157@tcp (at 192.168.204.157@tcp)
[ 2826.199383] Lustre: Skipped 14 previous similar messages
[ 2833.785923] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2877.343700] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 192.168.204.157@tcp) was lost; in progress operations using this service will fail
[ 2877.362028] LustreError: Skipped 3 previous similar messages
[ 2879.218978] Lustre: 15950:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.157@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2893.066995] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2948.461350] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2999.135307] Lustre: 2366:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789500679/real 1789500679]  req@ffff8a8f10773100 x1876424438691968/t0(0) o400->MGC192.168.204.157@tcp@192.168.204.157@tcp:26/25 lens 224/224 e 0 to 1 dl 1789500695 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2999.190910] Lustre: 2366:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
[ 3034.860023] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 15:32:09 (1789500729)
[ 3088.171339] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3090.585772] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3380.098726] LustreError: lustre-OST0000-osc-ffff8a8f08cae800: operation ldlm_enqueue to node 192.168.204.157@tcp failed: rc = -107
[ 3380.107071] LustreError: Skipped 9 previous similar messages
[ 3426.696836] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3429.428735] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3713.597076] Lustre: lustre-OST0000-osc-ffff8a8f08cae800: Connection to lustre-OST0000 (at 192.168.204.157@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3713.618242] Lustre: Skipped 5 previous similar messages
[ 3737.272964] Lustre: lustre-OST0000-osc-ffff8a8f08cae800: Connection restored to 192.168.204.157@tcp (at 192.168.204.157@tcp)
[ 3737.287535] Lustre: Skipped 8 previous similar messages
[ 3754.656566] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3756.563862] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4009.701900] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 15:48:24 (1789501704)
[ 4027.871155] Lustre: 51180:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789501707/real 1789501707]  req@ffff8a8f1076a300 x1876424456196224/t0(0) o101->lustre-MDT0000-mdc-ffff8a8f08cae800@192.168.204.157@tcp:12/10 lens 576/1152 e 0 to 1 dl 1789501723 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 4036.238718] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 15:48:51 (1789501731)
[ 4061.579702] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 15:49:16 (1789501756)
[ 4086.498267] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 15:49:41 (1789501781)
[ 4086.877467] Lustre: Mounted lustre-client - version 2.17.58_39_g104cf8c
[ 4118.495870] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 192.168.204.157@tcp) was lost; in progress operations using this service will fail
[ 4118.511500] LustreError: Skipped 2 previous similar messages
[ 4128.774713] Lustre: Evicted from MGS (at 192.168.204.157@tcp) after server handle changed from 0x3424367dca988ed9 to 0x3424367dcaad141a
[ 4128.787497] Lustre: Skipped 3 previous similar messages
[ 4134.965201] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4136.685900] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4138.674215] Lustre: Unmounted lustre-client
[ 4146.920357] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 15:50:41 (1789501841)
[ 4316.063597] Lustre: lustre-OST0000-osc-ffff8a8f08cae800: Connection to lustre-OST0000 (at 192.168.204.157@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4316.088111] Lustre: Skipped 14 previous similar messages
[ 4347.890468] Lustre: lustre-OST0000-osc-ffff8a8f08cae800: Connection restored to 192.168.204.157@tcp (at 192.168.204.157@tcp)
[ 4347.907903] Lustre: Skipped 17 previous similar messages
[ 4443.062200] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 15:55:37 (1789502137)
[ 4492.916548] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 15:56:27 (1789502187)
[ 4496.398023] Lustre: Unmounted lustre-client
[ 4517.620540] Lustre: Mounted lustre-client - version 2.17.58_39_g104cf8c
[ 4523.648132] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 15:56:58 (1789502218)
[ 4523.955333] LustreError: 57109:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4525.013572] LustreError: 57109:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4525.297299] Lustre: *** cfs_fail_loc=305, val=0***
[ 4550.038519] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 15:57:24 (1789502244)
[ 4550.562889] Lustre: Mounted lustre-client - version 2.17.58_39_g104cf8c
[ 4553.035343] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4563.359197] Lustre: Unmounted lustre-client
[ 4570.226315] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 15:57:45 (1789502265)
[ 4658.361227] LustreError: lustre-MDT0000-mdc-ffff8a8f07890800: operation mds_reint to node 192.168.204.157@tcp failed: rc = -19
[ 4658.366862] LustreError: Skipped 1 previous similar message
[ 4675.551737] Lustre: 2365:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789502355/real 1789502355]  req@ffff8a8f06141880 x1876424458923264/t0(0) o400->MGC192.168.204.157@tcp@192.168.204.157@tcp:26/25 lens 224/224 e 0 to 1 dl 1789502371 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4675.589885] Lustre: 2365:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 138 previous similar messages
[ 4675.596784] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 192.168.204.157@tcp) was lost; in progress operations using this service will fail
[ 4685.800541] Lustre: Evicted from MGS (at 192.168.204.157@tcp) after server handle changed from 0x3424367dcaad2ccb to 0x3424367dcaafe0cf
[ 4852.106781] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 16:02:26 (1789502546)
[ 4854.846421] Lustre: DEBUG MARKER: SKIP: recovery-small test_61 don't have skip orphan feature
[ 4856.697496] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 16:02:31 (1789502551)
[ 4857.479606] Lustre: Mounted lustre-client - version 2.17.58_39_g104cf8c
[ 4864.740276] LustreError: lustre-OST0000-osc-ffff8a8f07890800: operation ldlm_enqueue to node 192.168.204.157@tcp failed: rc = -107
[ 4864.765182] LustreError: lustre-OST0000-osc-ffff8a8f07890800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4864.791327] LustreError: 60122:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff8a8f07890800: namespace resource [0x240000400:0x3de2:0x0].0x0 (ffff8a8f060ebe00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4875.347619] Lustre: Unmounted lustre-client
[ 4881.021943] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 16:02:55 (1789502575)
[ 4888.555798] LustreError: lustre-MDT0000-mdc-ffff8a8f07890800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4888.593070] LustreError: 60794:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4898.324773] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 16:03:12 (1789502592)
[ 4898.581575] LustreError: 61494:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4902.665785] LustreError: lustre-MDT0000-mdc-ffff8a8f07890800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4902.677834] LustreError: 61515:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4902.696320] LustreError: 61494:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=900
[ 4902.700768] LustreError: 61494:0:(import.c:762:ptlrpc_connect_import_locked()) already connecting
[ 4902.764563] LustreError: 61520:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4903.870812] LustreError: 61527:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4903.889117] LustreError: 61527:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4906.232981] LustreError: 61549:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4906.240983] LustreError: 61549:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4910.755347] LustreError: 61594:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4910.761104] LustreError: 61594:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4921.283756] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 16:03:35 (1789502615)
[ 4928.487334] Lustre: lustre-OST0000-osc-ffff8a8f07890800: Connection to lustre-OST0000 (at 192.168.204.157@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4928.510548] Lustre: Skipped 16 previous similar messages
[ 4960.393297] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4961.851732] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4972.360241] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 16:04:26 (1789502666)
[ 4997.023851] Lustre: lustre-OST0000-osc-ffff8a8f07890800: Connection restored to 192.168.204.157@tcp (at 192.168.204.157@tcp)
[ 4997.035013] Lustre: Skipped 14 previous similar messages
[ 5011.042651] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5012.460754] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5023.102988] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 16:05:17 (1789502717)
[ 5086.388327] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5088.050259] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5096.528225] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 16:06:31 (1789502791)
[ 5135.365750] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5137.204764] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5143.563202] Lustre: Unmounted lustre-client
[ 5197.664182] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5200.223811] Lustre: Mounted lustre-client - version 2.17.58_39_g104cf8c
[ 5207.156150] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 16:08:21 (1789502901)
[ 5209.465308] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5211.123265] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 16:08:25 (1789502905)
[ 5241.540287] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 16:08:56 (1789502936)
[ 5243.133634] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5244.984958] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 16:08:59 (1789502939)
[ 5246.369151] Lustre: *** cfs_fail_loc=805, val=0***
[ 5246.415866] Lustre: Mounted lustre-client - version 2.17.58_39_g104cf8c
[ 5251.711381] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5272.048335] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 192.168.204.157@tcp) was lost; in progress operations using this service will fail
[ 5272.088277] Lustre: Evicted from MGS (at 192.168.204.157@tcp) after server handle changed from 0x3424367dcab4b0e4 to 0x3424367dcab4b2ff
[ 5273.964181] Lustre: 68175:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.157@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 5277.151252] Lustre: 2366:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789502957/real 1789502957]  req@ffff8a8f055cb480 x1876424460963840/t0(0) o400->lustre-MDT0000-mdc-ffff8a8f1075d000@192.168.204.157@tcp:12/10 lens 224/224 e 0 to 1 dl 1789502973 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5277.201531] Lustre: 2366:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 5291.079172] Lustre: Unmounted lustre-client
[ 5300.266529] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 16:09:54 (1789502994)
[ 5325.940519] Lustre: 68175:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.157@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 5342.759707] LustreError: 2364:0:(client.c:3449:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff8a8f046fd180 x1876424460975744/t85899345925(85899345925) o101->lustre-MDT0000-mdc-ffff8a8f1075d000@192.168.204.157@tcp:12/10 lens 648/608 e 0 to 0 dl 1789503054 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5348.020448] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5349.596845] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5357.720809] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 16:10:52 (1789503052)
[ 5359.320204] LustreError: lustre-OST0000-osc-ffff8a8f1075d000: operation ost_write to node 192.168.204.157@tcp failed: rc = -107
[ 5359.333079] LustreError: Skipped 2 previous similar messages
[ 5359.355681] LustreError: lustre-OST0000-osc-ffff8a8f1075d000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5359.379239] Lustre: 2366:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.157@tcp:/lustre/fid: [0x200009872:0x6:0x0]// may get corrupted (rc -5)
[ 5359.413567] LustreError: 72723:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff8a8f1075d000: namespace resource [0x240000400:0x3e42:0x0].0x0 (ffff8a8f057b3a00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5367.646781] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 16:11:02 (1789503062)
[ 5369.730808] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5371.351782] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 16:11:06 (1789503066)
[ 5372.858443] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5374.793091] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 16:11:09 (1789503069)
[ 5376.489973] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5378.290277] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 16:11:13 (1789503073)
[ 5379.834667] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5381.950901] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 16:11:16 (1789503076)
[ 5383.453355] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5385.100626] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 16:11:19 (1789503079)
[ 5386.311382] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5387.920282] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 16:11:22 (1789503082)
[ 5390.240176] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5391.969936] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 16:11:26 (1789503086)
[ 5393.758466] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5396.113280] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 16:11:30 (1789503090)
[ 5398.147830] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5399.828557] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 16:11:34 (1789503094)
[ 5401.474667] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5403.147817] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 16:11:37 (1789503097)
[ 5404.881542] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5406.691602] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 16:11:41 (1789503101)
[ 5409.803694] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5412.124501] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 16:11:46 (1789503106)
[ 5435.872268] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 192.168.204.157@tcp) was lost; in progress operations using this service will fail
[ 5435.892942] LustreError: Skipped 1 previous similar message
[ 5446.142265] Lustre: Evicted from MGS (at 192.168.204.157@tcp) after server handle changed from 0x3424367dcab4b7dd to 0x3424367dcab4bf62
[ 5446.171660] Lustre: Skipped 1 previous similar message
[ 5454.148534] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 16:12:28 (1789503148)
[ 5484.500367] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 16:12:58 (1789503178)
[ 5485.157411] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5496.898532] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 16:13:11 (1789503191)
[ 5497.632917] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5510.610211] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 16:13:25 (1789503205)
[ 5510.939472] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5521.381218] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 16:13:35 (1789503215)
[ 5521.774348] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5531.448361] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 16:13:46 (1789503226)
[ 5531.805955] Lustre: *** cfs_fail_loc=510, val=3***
[ 5541.390459] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 16:13:56 (1789503236)
[ 5541.878966] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5552.774270] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 16:14:07 (1789503247)
[ 5553.137808] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5611.999310] Lustre: lustre-OST0000-osc-ffff8a8f1075d000: Connection to lustre-OST0000 (at 192.168.204.157@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5612.042941] Lustre: Skipped 9 previous similar messages
[ 5620.458551] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 16:15:14 (1789503314)
[ 5620.957943] LustreError: 82931:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5623.860959] LustreError: lustre-MDT0000-mdc-ffff8a8f1075d000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5623.873614] LustreError: 82946:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8a8f1075d000: inode [0x200009872:0x10:0x0] mdc close failed: rc = -108
[ 5623.892439] LustreError: 82946:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8a8f1075d000: namespace resource [0x200009872:0x11:0x0].0xc (ffff8a8f06227800) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5623.907481] Lustre: lustre-MDT0000-mdc-ffff8a8f1075d000: Connection restored to 192.168.204.157@tcp (at 192.168.204.157@tcp)
[ 5623.915512] Lustre: Skipped 10 previous similar messages
[ 5625.031157] LustreError: 82931:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5627.166458] LustreError: 82961:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5630.448556] LustreError: lustre-MDT0000-mdc-ffff8a8f1075d000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5630.500616] LustreError: 82976:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8a8f1075d000: namespace resource [0x200009872:0x11:0x0].0xc (ffff8a8f0575b800) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5630.547208] LustreError: 82976:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5631.255257] LustreError: 82961:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5638.758261] LustreError: lustre-MDT0000-mdc-ffff8a8f1075d000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5639.235851] LustreError: 83017:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5642.092783] LustreError: lustre-MDT0000-mdc-ffff8a8f1075d000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5642.205615] LustreError: 83037:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5642.235447] LustreError: 83037:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 20 previous similar messages
[ 5643.311106] LustreError: 83017:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5643.331968] LustreError: 83017:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8a8f1075d000: inode [0x200009872:0x11:0x0] mdc close failed: rc = -108
[ 5643.336446] LustreError: 83043:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5643.394297] LustreError: 83043:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5645.708556] LustreError: 83066:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5645.717047] LustreError: 83066:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5649.247178] LustreError: 83092:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5652.446841] LustreError: lustre-MDT0000-mdc-ffff8a8f1075d000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5652.483559] LustreError: 83107:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8a8f1075d000: namespace resource [0x200009872:0x11:0x0].0xc (ffff8a8f060eb600) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5652.509928] LustreError: 83107:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5653.247200] LustreError: 83092:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5660.571910] LustreError: lustre-MDT0000-mdc-ffff8a8f1075d000: operation ldlm_enqueue to node 192.168.204.157@tcp failed: rc = -107
[ 5660.583940] LustreError: Skipped 5 previous similar messages
[ 5660.597721] LustreError: lustre-MDT0000-mdc-ffff8a8f1075d000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5660.615194] LustreError: 83136:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8a8f1075d000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff8a8f06227300) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5660.637892] LustreError: 83136:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5661.295181] LustreError: 83149:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5664.738797] LustreError: lustre-MDT0000-mdc-ffff8a8f1075d000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5665.383130] LustreError: 83149:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5668.352292] LustreError: lustre-MDT0000-mdc-ffff8a8f1075d000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5668.459356] LustreError: 83196:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5668.476707] LustreError: 83196:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5669.559737] LustreError: 83177:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8a8f1075d000: inode [0x200009872:0x11:0x0] mdc close failed: rc = -108
[ 5677.810727] LustreError: 83248:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5677.823847] LustreError: 83248:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5681.911172] LustreError: 83248:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5681.929290] LustreError: 83248:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5681.976577] LustreError: lustre-MDT0000-mdc-ffff8a8f1075d000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5682.004500] LustreError: 83248:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8a8f1075d000: inode [0x200009872:0x11:0x0] mdc close failed: rc = -5
[ 5693.673208] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 16:16:27 (1789503387)
[ 5715.045627] LustreError: 68172:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5719.130561] LustreError: 68172:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5727.440207] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 16:17:01 (1789503421)
[ 5778.211111] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 16:17:52 (1789503472)
[ 5842.006671] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 16:18:57 (1789503537)
[ 5877.022646] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 16:19:31 (1789503571)
[ 5877.500870] Lustre: Mounted lustre-client - version 2.17.58_39_g104cf8c
[ 6000.854173] Lustre: Unmounted lustre-client
[ 6007.223679] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 16:21:42 (1789503702)
[ 6007.862750] LustreError: 87531:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 6010.631463] LustreError: lustre-OST0000-osc-ffff8a8f1075d000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6010.644083] LustreError: 87618:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff8a8f1075d000: namespace resource [0x240000400:0x3e6d:0x0].0x0 (ffff8a8f060eb600) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6010.669399] LustreError: 87618:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 2 previous similar messages
[ 6010.823128] LustreError: 87531:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout interrupted
[ 6010.832387] Lustre: 2366:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.157@tcp:/lustre/fid: [0x20000b7b9:0x9:0x0]// may get corrupted (rc -108)
[ 6018.839255] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 16:21:53 (1789503713)
[ 6077.407359] Lustre: 88198:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789503718/real 1789503718]  req@ffff8a8f056b1f80 x1876424461142528/t0(0) o101->lustre-MDT0000-mdc-ffff8a8f1075d000@192.168.204.157@tcp:12/10 lens 328/344 e 0 to 1 dl 1789503773 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 6077.444034] Lustre: 88198:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 18 previous similar messages
[ 6085.021783] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 16:22:59 (1789503779)
[ 6086.652170] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 6088.790709] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 16:23:03 (1789503783)
[ 6153.142880] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6155.030994] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 16:24:09 (1789503849)
[ 6213.276090] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 16:25:07 (1789503907)
[ 6216.005830] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6218.071166] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 16:25:12 (1789503912)
[ 6219.735538] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6221.386817] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 16:25:16 (1789503916)
[ 6263.183015] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 16:25:57 (1789503957)
[ 6276.777779] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6304.479207] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 192.168.204.157@tcp) was lost; in progress operations using this service will fail
[ 6310.751254] Lustre: lustre-MDT0000-mdc-ffff8a8f1075d000: Connection to lustre-MDT0000 (at 192.168.204.157@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6310.769586] Lustre: Skipped 16 previous similar messages
[ 6314.854080] Lustre: Evicted from MGS (at 192.168.204.157@tcp) after server handle changed from 0x3424367dcab4bf62 to 0x3424367dcab4d75d
[ 6314.866523] Lustre: MGC192.168.204.157@tcp: Connection restored to 192.168.204.157@tcp (at 192.168.204.157@tcp)
[ 6314.879842] Lustre: Skipped 15 previous similar messages
[ 6331.939846] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6333.998929] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6346.596667] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 16:27:21 (1789504041)
[ 6349.569086] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6351.801764] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 16:27:26 (1789504046)
[ 6366.198442] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 192.168.204.157@tcp) was lost; in progress operations using this service will fail
[ 6366.248816] Lustre: Evicted from MGS (at 192.168.204.157@tcp) after server handle changed from 0x3424367dcab4d75d to 0x3424367dcab4db93
[ 6383.852981] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 16:27:58 (1789504078)
[ 6438.690927] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 16:28:52 (1789504132)
[ 6446.742707] LustreError: lustre-OST0000-osc-ffff8a8f1075d000: operation ost_setattr to node 192.168.204.157@tcp failed: rc = -19
[ 6446.759528] LustreError: Skipped 3 previous similar messages
[ 6501.758825] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6504.755194] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6591.455218] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 192.168.204.157@tcp) was lost; in progress operations using this service will fail
[ 6591.467935] LustreError: Skipped 1 previous similar message
[ 6601.732153] Lustre: Evicted from MGS (at 192.168.204.157@tcp) after server handle changed from 0x3424367dcab4de1e to 0x3424367dcab4ed36
[ 6601.750982] Lustre: Skipped 1 previous similar message
[ 6618.372220] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6620.057121] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6669.360672] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6671.353709] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6687.711864] Lustre: 2367:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789504323/real 1789504323]  req@ffff8a8f32611180 x1876424467670528/t0(0) o400->lustre-MDT0000-mdc-ffff8a8f1075d000@192.168.204.157@tcp:12/10 lens 224/224 e 0 to 1 dl 1789504383 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6687.738061] Lustre: 2367:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 24 previous similar messages
[ 6819.985515] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 16:35:14 (1789504514)
[ 6885.451199] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6890.046874] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 7144.707484] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 16:40:39 (1789504839)
[ 7298.024356] Lustre: lustre-MDT0000-mdc-ffff8a8f1075d000: Connection to lustre-MDT0000 (at 192.168.204.157@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7298.049624] Lustre: Skipped 6 previous similar messages
[ 7298.062641] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 192.168.204.157@tcp) was lost; in progress operations using this service will fail
[ 7298.077587] LustreError: Skipped 1 previous similar message
[ 7298.125512] Lustre: Evicted from MGS (at 192.168.204.157@tcp) after server handle changed from 0x3424367dcab4f077 to 0x3424367dcac49477
[ 7298.145472] Lustre: Skipped 1 previous similar message
[ 7298.158699] Lustre: MGC192.168.204.157@tcp: Connection restored to 192.168.204.157@tcp (at 192.168.204.157@tcp)
[ 7298.176221] Lustre: Skipped 9 previous similar messages
[ 7325.664158] Lustre: 2368:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789504983/real 1789504983]  req@ffff8a8f2da7ed80 x1876424478848768/t0(0) o400->lustre-MDT0000-mdc-ffff8a8f1075d000@192.168.204.157@tcp:12/10 lens 224/224 e 0 to 1 dl 1789505021 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7325.690644] Lustre: 2368:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
[ 7331.154539] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 16:43:45 (1789505025)
[ 7332.590366] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7334.242322] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 16:43:49 (1789505029)
[ 7336.135825] LustreError: lustre-MDT0000-mdc-ffff8a8f1075d000: operation ldlm_enqueue to node 192.168.204.157@tcp failed: rc = -107
[ 7336.151217] LustreError: Skipped 189 previous similar messages
[ 7336.174318] LustreError: lustre-MDT0000-mdc-ffff8a8f1075d000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7343.816436] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 16:43:58 (1789505038)
[ 7511.015894] LustreError: lustre-OST0000-osc-ffff8a8f1075d000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7512.674582] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 16:46:47 (1789505207)
[ 7552.341911] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 16:47:27 (1789505247)
[ 7553.428743] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7554.713739] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 16:47:29 (1789505249)
[ 7555.759928] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7557.150299] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 16:47:32 (1789505252)
[ 7588.242102] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 16:48:03 (1789505283)
[ 7624.687284] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 192.168.204.157@tcp) was lost; in progress operations using this service will fail
[ 7624.707052] Lustre: Evicted from MGS (at 192.168.204.157@tcp) after server handle changed from 0x3424367dcac49477 to 0x3424367dcac4bc40
[ 7624.722398] Lustre: Skipped 1 previous similar message
[ 7638.087823] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 16:48:53 (1789505333)
[ 7639.427922] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7641.121678] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 16:48:55 (1789505335)
[ 7642.277668] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7643.812712] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 16:48:58 (1789505338)
[ 7652.401594] Lustre: Unmounted lustre-client
[ 7653.966404] Lustre: Mounted lustre-client - version 2.17.58_39_g104cf8c
[ 7657.090092] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7681.010783] Lustre: 106655:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.157@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 7701.128644] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2