[ 1933.804318] LustreError: lustre-OST0000-osc-ffffa0eb5192f000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1933.824338] Lustre: 2360:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.139@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1942.197394] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 20:37:17 (1787099837)
[ 1945.179826] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1947.278852] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 20:37:23 (1787099843)
[ 1949.613433] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1951.369563] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 20:37:27 (1787099847)
[ 1955.044221] LustreError: lustre-MDT0000-mdc-ffffa0eb5192f000: operation mds_close to node 192.168.202.139@tcp failed: rc = -19
[ 1955.060914] LustreError: Skipped 2 previous similar messages
[ 1971.680472] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 192.168.202.139@tcp) was lost; in progress operations using this service will fail
[ 1981.871388] Lustre: Evicted from MGS (at 192.168.202.139@tcp) after server handle changed from 0x5e599baba9cda0bc to 0x5e599baba9cdb990
[ 2090.002293] LustreError: lustre-MDT0000-mdc-ffffa0eb5192f000: operation mds_close to node 192.168.202.139@tcp failed: rc = -19
[ 2090.013557] LustreError: Skipped 1 previous similar message
[ 2107.876435] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 192.168.202.139@tcp) was lost; in progress operations using this service will fail
[ 2118.134917] Lustre: Evicted from MGS (at 192.168.202.139@tcp) after server handle changed from 0x5e599baba9cdb990 to 0x5e599baba9cf807b
[ 2138.566327] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 20:40:34 (1787100034)
[ 2139.202972] Lustre: *** cfs_fail_loc=305, val=0***
[ 2139.208464] Lustre: Skipped 2 previous similar messages
[ 2178.528474] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 192.168.202.139@tcp) was lost; in progress operations using this service will fail
[ 2188.790266] Lustre: Evicted from MGS (at 192.168.202.139@tcp) after server handle changed from 0x5e599baba9cf807b to 0x5e599baba9cf8322
[ 2188.804908] Lustre: MGC192.168.202.139@tcp: Connection restored to 192.168.202.139@tcp (at 192.168.202.139@tcp)
[ 2188.818931] Lustre: Skipped 13 previous similar messages
[ 2191.625187] Lustre: lustre-MDT0000-mdc-ffffa0eb5192f000: Connection to lustre-MDT0000 (at 192.168.202.139@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2191.637981] Lustre: Skipped 11 previous similar messages
[ 2205.862934] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2207.844537] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2218.297943] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 20:41:54 (1787100114)
[ 2234.908896] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 192.168.202.139@tcp) was lost; in progress operations using this service will fail
[ 2235.006805] Lustre: Evicted from MGS (at 192.168.202.139@tcp) after server handle changed from 0x5e599baba9cf8322 to 0x5e599baba9cf891f
[ 2235.153753] LustreError: lustre-MDT0000-mdc-ffffa0eb5192f000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2269.608764] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 20:42:46 (1787100166)
[ 2287.722173] LustreError: lustre-OST0000-osc-ffffa0eb5192f000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2306.684676] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 20:43:22 (1787100202)
[ 2320.005047] LustreError: lustre-MDT0000-mdc-ffffa0eb5192f000: operation ldlm_enqueue to node 192.168.202.139@tcp failed: rc = -19
[ 2337.825027] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 192.168.202.139@tcp) was lost; in progress operations using this service will fail
[ 2348.010910] Lustre: Evicted from MGS (at 192.168.202.139@tcp) after server handle changed from 0x5e599baba9cf891f to 0x5e599baba9cfdf25
[ 2366.523289] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2370.234321] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2454.502481] Lustre: 2361:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787100336/real 1787100336]  req@ffffa0eb47f50700 x1873908075992832/t0(0) o400->MGC192.168.202.139@tcp@192.168.202.139@tcp:26/25 lens 224/224 e 0 to 1 dl 1787100352 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2454.525755] Lustre: 2361:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 17 previous similar messages
[ 2454.531136] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 192.168.202.139@tcp) was lost; in progress operations using this service will fail
[ 2464.756038] Lustre: Evicted from MGS (at 192.168.202.139@tcp) after server handle changed from 0x5e599baba9cfdf25 to 0x5e599baba9d15b8d
[ 2488.196660] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2491.312029] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2558.401430] LustreError: lustre-MDT0000-mdc-ffffa0eb5192f000: operation mds_reint to node 192.168.202.139@tcp failed: rc = -19
[ 2558.409231] LustreError: Skipped 2 previous similar messages
[ 2577.376314] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 192.168.202.139@tcp) was lost; in progress operations using this service will fail
[ 2586.554960] Lustre: Evicted from MGS (at 192.168.202.139@tcp) after server handle changed from 0x5e599baba9d15b8d to 0x5e599baba9d2ac27
[ 2607.614226] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2610.354964] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2640.264740] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 20:48:55 (1787100535)
[ 2663.392373] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 192.168.202.139@tcp) was lost; in progress operations using this service will fail
[ 2673.673728] Lustre: Evicted from MGS (at 192.168.202.139@tcp) after server handle changed from 0x5e599baba9d2ac27 to 0x5e599baba9d39f36
[ 2675.357430] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2718.725495] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2764.944357] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2798.560571] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 192.168.202.139@tcp) was lost; in progress operations using this service will fail
[ 2798.574668] LustreError: Skipped 2 previous similar messages
[ 2808.816724] Lustre: Evicted from MGS (at 192.168.202.139@tcp) after server handle changed from 0x5e599baba9d4012f to 0x5e599baba9d450ac
[ 2808.833718] Lustre: Skipped 2 previous similar messages
[ 2808.886222] Lustre: MGC192.168.202.139@tcp: Connection restored to 192.168.202.139@tcp (at 192.168.202.139@tcp)
[ 2808.903820] Lustre: Skipped 15 previous similar messages
[ 2820.071957] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2843.187305] LustreError: lustre-MDT0000-mdc-ffffa0eb5192f000: operation ldlm_enqueue to node 192.168.202.139@tcp failed: rc = -19
[ 2843.210205] LustreError: Skipped 5 previous similar messages
[ 2843.214767] Lustre: lustre-MDT0000-mdc-ffffa0eb5192f000: Connection to lustre-MDT0000 (at 192.168.202.139@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2843.239209] Lustre: Skipped 8 previous similar messages
[ 2872.122957] Lustre: 15924:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.202.139@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2884.144172] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2950.447654] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 3017.910949] Lustre: 15924:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.202.139@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 3052.506607] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 20:55:48 (1787100948)
[ 3100.583179] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3102.490869] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3395.800764] LustreError: lustre-OST0000-osc-ffffa0eb5192f000: operation ost_write to node 192.168.202.139@tcp failed: rc = -19
[ 3395.809763] LustreError: Skipped 4 previous similar messages
[ 3415.534240] Lustre: lustre-OST0000-osc-ffffa0eb5192f000: Connection restored to 192.168.202.139@tcp (at 192.168.202.139@tcp)
[ 3415.543149] Lustre: Skipped 8 previous similar messages
[ 3426.414037] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3427.825438] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3727.615348] Lustre: lustre-OST0000-osc-ffffa0eb5192f000: Connection to lustre-OST0000 (at 192.168.202.139@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3727.631054] Lustre: Skipped 4 previous similar messages
[ 3757.448210] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3758.727829] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4020.526159] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 21:11:57 (1787101917)
[ 4036.576198] Lustre: 50555:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787101918/real 1787101918]  req@ffffa0eb50904a80 x1873908108548352/t0(0) o101->lustre-MDT0000-mdc-ffffa0eb5192f000@192.168.202.139@tcp:12/10 lens 576/1152 e 0 to 1 dl 1787101934 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 4036.591118] Lustre: 50555:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
[ 4036.607206] Lustre: lustre-MDT0000-mdc-ffffa0eb5192f000: Connection restored to 192.168.202.139@tcp (at 192.168.202.139@tcp)
[ 4036.610898] Lustre: Skipped 1 previous similar message
[ 4040.201811] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 21:12:17 (1787101937)
[ 4061.065227] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 21:12:37 (1787101957)
[ 4082.208177] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 21:12:59 (1787101979)
[ 4082.394880] Lustre: Mounted lustre-client - version 2.17.57_1_g147bb23
[ 4108.261222] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 192.168.202.139@tcp) was lost; in progress operations using this service will fail
[ 4108.268536] LustreError: Skipped 3 previous similar messages
[ 4108.278844] Lustre: Evicted from MGS (at 192.168.202.139@tcp) after server handle changed from 0x5e599baba9d5f283 to 0x5e599baba9f77986
[ 4108.286845] Lustre: Skipped 3 previous similar messages
[ 4114.400157] Lustre: 2361:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787101995/real 1787101995]  req@ffffa0eb51228000 x1873908108566016/t0(0) o400->lustre-MDT0000-mdc-ffffa0eb5192f000@192.168.202.139@tcp:12/10 lens 224/224 e 0 to 1 dl 1787102011 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4114.413198] Lustre: 2361:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 4115.655404] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4116.445810] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4117.972731] Lustre: Unmounted lustre-client
[ 4120.913166] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 21:13:37 (1787102017)
[ 4269.984343] Lustre: 2362:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787102151/real 1787102151]  req@ffffa0eb51229f80 x1873908108597120/t0(0) o4->lustre-OST0000-osc-ffffa0eb5192f000@192.168.202.139@tcp:6/4 lens 488/448 e 0 to 1 dl 1787102167 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4269.997385] Lustre: 2362:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 71 previous similar messages
[ 4334.560287] Lustre: lustre-OST0000-osc-ffffa0eb5192f000: Connection to lustre-OST0000 (at 192.168.202.139@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4334.566392] Lustre: Skipped 17 previous similar messages
[ 4404.982811] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 21:18:21 (1787102301)
[ 4448.762553] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 21:19:05 (1787102345)
[ 4450.157697] Lustre: Unmounted lustre-client
[ 4459.603820] Lustre: Mounted lustre-client - version 2.17.57_1_g147bb23
[ 4462.121713] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 21:19:19 (1787102359)
[ 4462.234971] LustreError: 56464:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4463.280064] LustreError: 56464:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4463.315942] Lustre: *** cfs_fail_loc=305, val=0***
[ 4482.667444] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 21:19:39 (1787102379)
[ 4482.889420] Lustre: Mounted lustre-client - version 2.17.57_1_g147bb23
[ 4483.489540] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4493.755224] Lustre: Unmounted lustre-client
[ 4496.688609] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 21:19:53 (1787102393)
[ 4538.574225] LustreError: lustre-MDT0000-mdc-ffffa0eb50822800: operation mds_reint to node 192.168.202.139@tcp failed: rc = -19
[ 4538.577581] LustreError: Skipped 1 previous similar message
[ 4554.722271] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 192.168.202.139@tcp) was lost; in progress operations using this service will fail
[ 4554.729474] Lustre: Evicted from MGS (at 192.168.202.139@tcp) after server handle changed from 0x5e599baba9f79c9b to 0x5e599baba9fa709d
[ 4606.755253] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 21:21:43 (1787102503)
[ 4608.836644] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4616.190890] LustreError: lustre-MDT0000-mdc-ffffa0eb50822800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4621.956510] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 21:21:58 (1787102518)
[ 4622.109387] Lustre: Mounted lustre-client - version 2.17.57_1_g147bb23
[ 4627.434260] LustreError: lustre-OST0000-osc-ffffa0eb50822800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4627.443786] LustreError: 60149:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffffa0eb50822800: namespace resource [0x240000400:0x53c3:0x0].0x0 (ffffa0eb46c35900) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4638.688199] Lustre: 2360:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787102520/real 1787102520]  req@ffffa0eb441a7480 x1873908113214976/t0(0) o103->lustre-OST0000-osc-ffffa0eb51781800@192.168.202.139@tcp:17/18 lens 328/224 e 0 to 1 dl 1787102536 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295
[ 4638.697421] Lustre: 2360:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 76 previous similar messages
[ 4638.709106] Lustre: lustre-OST0000-osc-ffffa0eb51781800: Connection restored to 192.168.202.139@tcp (at 192.168.202.139@tcp)
[ 4638.712495] Lustre: Skipped 28 previous similar messages
[ 4639.058191] Lustre: Unmounted lustre-client
[ 4641.509644] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 21:22:18 (1787102538)
[ 4646.437034] LustreError: lustre-MDT0000-mdc-ffffa0eb50822800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4646.444071] LustreError: 60820:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4649.314345] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 21:22:26 (1787102546)
[ 4649.392716] LustreError: 61523:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4654.362125] LustreError: lustre-MDT0000-mdc-ffffa0eb50822800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4654.366957] LustreError: 61539:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4654.369634] LustreError: 61523:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=25
[ 4654.372825] LustreError: 61523:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting
[ 4654.382927] LustreError: 61544:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4655.414346] LustreError: 61551:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4655.418145] LustreError: 61551:0:(file.c:6156:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4657.518697] LustreError: 61573:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4657.523072] LustreError: 61573:0:(file.c:6156:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4661.733552] LustreError: 61618:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4661.737736] LustreError: 61618:0:(file.c:6156:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4667.908251] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 21:22:44 (1787102564)
[ 4701.961589] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4703.298772] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4713.068833] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 21:23:29 (1787102609)
[ 4743.788561] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4744.828985] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4751.624485] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 21:24:08 (1787102648)
[ 4803.697505] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4804.681746] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4809.755949] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 21:25:06 (1787102706)
[ 4835.453662] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4836.182609] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4839.208176] Lustre: Unmounted lustre-client
[ 4874.315925] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4876.341656] Lustre: Mounted lustre-client - version 2.17.57_1_g147bb23
[ 4881.145254] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 21:26:17 (1787102777)
[ 4882.696399] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 4883.677823] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 21:26:20 (1787102780)
[ 4902.450856] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 21:26:39 (1787102799)
[ 4903.257670] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 4904.295245] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 21:26:41 (1787102801)
[ 4905.123062] Lustre: *** cfs_fail_loc=805, val=0***
[ 4905.170349] Lustre: Mounted lustre-client - version 2.17.57_1_g147bb23
[ 4908.593374] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4925.939636] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 192.168.202.139@tcp) was lost; in progress operations using this service will fail
[ 4925.950968] LustreError: Skipped 1 previous similar message
[ 4925.966566] Lustre: Evicted from MGS (at 192.168.202.139@tcp) after server handle changed from 0x5e599baba9ff0e4d to 0x5e599baba9ff1164
[ 4925.975940] Lustre: Skipped 1 previous similar message
[ 4931.615441] Lustre: Unmounted lustre-client
[ 4936.221133] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 21:27:13 (1787102833)
[ 4953.056365] Lustre: lustre-MDT0000-mdc-ffffa0eb5192f000: Connection to lustre-MDT0000 (at 192.168.202.139@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4953.065876] Lustre: Skipped 21 previous similar messages
[ 4953.115638] LustreError: 2358:0:(client.c:3435:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffffa0eb45f70a80 x1873908113288960/t90194313220(90194313220) o101->lustre-MDT0000-mdc-ffffa0eb5192f000@192.168.202.139@tcp:12/10 lens 664/608 e 0 to 0 dl 1787102866 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 4960.727188] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4961.581626] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4966.232444] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 21:27:43 (1787102863)
[ 4966.944137] LustreError: lustre-OST0000-osc-ffffa0eb5192f000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4966.955047] Lustre: 2361:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.139@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 4966.980677] LustreError: 72715:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffffa0eb5192f000: namespace resource [0x240000400:0x5422:0x0].0x0 (ffffa0eb443f5200) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4966.986644] LustreError: 72715:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4971.490031] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 21:27:48 (1787102868)
[ 4972.215955] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 4973.043313] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 21:27:49 (1787102869)
[ 4973.744368] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 4974.524970] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 21:27:51 (1787102871)
[ 4975.262759] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 4976.216309] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 21:27:53 (1787102873)
[ 4976.930773] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 4977.800361] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 21:27:54 (1787102874)
[ 4978.574607] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 4979.459544] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 21:27:56 (1787102876)
[ 4980.141319] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 4980.880436] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 21:27:57 (1787102877)
[ 4981.625662] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 4982.388938] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 21:27:59 (1787102879)
[ 4983.124527] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 4983.919930] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 21:28:00 (1787102880)
[ 4984.515869] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 4985.193210] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 21:28:02 (1787102882)
[ 4985.806628] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 4986.588443] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 21:28:03 (1787102883)
[ 4987.239619] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 4987.963607] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 21:28:04 (1787102884)
[ 4989.066880] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 4989.800402] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 21:28:06 (1787102886)
[ 5001.762567] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 21:28:18 (1787102898)
[ 5025.498937] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 21:28:42 (1787102922)
[ 5025.607098] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5031.853452] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 21:28:48 (1787102928)
[ 5031.951787] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5038.328184] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 21:28:55 (1787102935)
[ 5038.465396] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5042.290124] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 21:28:59 (1787102939)
[ 5042.369826] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5046.115786] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 21:29:03 (1787102943)
[ 5046.210379] Lustre: *** cfs_fail_loc=510, val=3***
[ 5050.187775] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 21:29:07 (1787102947)
[ 5050.294977] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5056.543994] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 21:29:13 (1787102953)
[ 5119.972513] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 21:30:17 (1787103017)
[ 5120.083911] LustreError: 82907:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5122.402680] LustreError: lustre-MDT0000-mdc-ffffa0eb5192f000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5122.407584] LustreError: 82921:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffffa0eb5192f000: inode [0x20000a042:0x8:0x0] mdc close failed: rc = -108
[ 5122.417345] LustreError: 82921:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffffa0eb5192f000: namespace resource [0x20000a042:0x11:0x0].0xc (ffffa0eb4779a800) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5124.144136] LustreError: 82907:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5126.174592] LustreError: 82937:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5128.507103] LustreError: lustre-MDT0000-mdc-ffffa0eb5192f000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5128.512519] LustreError: 82951:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffffa0eb5192f000: namespace resource [0x20000a042:0x11:0x0].0xc (ffffa0eb4779a100) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5128.517995] LustreError: 82951:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5130.232124] LustreError: 82937:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5136.526064] LustreError: lustre-MDT0000-mdc-ffffa0eb5192f000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5136.672701] LustreError: 82994:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5138.987864] LustreError: lustre-MDT0000-mdc-ffffa0eb5192f000: operation ldlm_enqueue to node 192.168.202.139@tcp failed: rc = -107
[ 5138.991494] LustreError: Skipped 6 previous similar messages
[ 5138.995360] LustreError: lustre-MDT0000-mdc-ffffa0eb5192f000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5139.016727] LustreError: 83013:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5139.020122] LustreError: 83013:0:(file.c:6156:ll_inode_revalidate_fini()) Skipped 20 previous similar messages
[ 5140.736179] LustreError: 82994:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5140.741109] LustreError: 82994:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffffa0eb5192f000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5140.744728] LustreError: 82994:0:(file.c:251:ll_close_inode_openhandle()) Skipped 4 previous similar messages
[ 5145.417268] LustreError: 83068:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5147.776813] LustreError: lustre-MDT0000-mdc-ffffa0eb5192f000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5147.787639] LustreError: 83083:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffffa0eb5192f000: namespace resource [0x20000a042:0x11:0x0].0xc (ffffa0eb48155800) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5147.794916] LustreError: 83083:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5149.480146] LustreError: 83068:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5155.793348] LustreError: lustre-MDT0000-mdc-ffffa0eb5192f000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5155.959454] LustreError: 83125:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5158.263707] LustreError: lustre-MDT0000-mdc-ffffa0eb5192f000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5158.269325] LustreError: 83140:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffffa0eb5192f000: namespace resource [0x20000a042:0x11:0x0].0xc (ffffa0eb4779a400) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5158.275242] LustreError: 83140:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5160.016145] LustreError: 83125:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5162.354445] LustreError: lustre-MDT0000-mdc-ffffa0eb5192f000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5162.377327] LustreError: 83172:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5162.380868] LustreError: 83172:0:(file.c:6156:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5164.112774] LustreError: 83153:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffffa0eb5192f000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5170.722628] LustreError: 83224:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5170.725559] LustreError: 83224:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5173.261945] LustreError: lustre-MDT0000-mdc-ffffa0eb5192f000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5174.784184] LustreError: 83224:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5174.787680] LustreError: 83224:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5174.791614] LustreError: 83224:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffffa0eb5192f000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5179.733366] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 21:31:16 (1787103076)
[ 5196.302102] LustreError: 68184:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5196.306692] LustreError: 68184:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5200.368132] LustreError: 68184:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5200.371091] LustreError: 68184:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5202.736635] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 21:31:39 (1787103099)
[ 5245.920187] Lustre: 84669:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787103101/real 1787103101]  req@ffffa0eb44297800 x1873908113396864/t0(0) o101->lustre-MDT0000-mdc-ffffa0eb5192f000@192.168.202.139@tcp:12/10 lens 576/1152 e 1 to 1 dl 1787103143 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'stat.0' uid:0 gid:0 projid:0
[ 5245.931054] Lustre: 84669:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 14 previous similar messages
[ 5245.944182] Lustre: lustre-MDT0000-mdc-ffffa0eb5192f000: Connection restored to 192.168.202.139@tcp (at 192.168.202.139@tcp)
[ 5245.948111] Lustre: Skipped 26 previous similar messages
[ 5248.196638] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 21:32:25 (1787103145)
[ 5308.367680] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 21:33:25 (1787103205)
[ 5335.352682] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 21:33:52 (1787103232)
[ 5335.570583] Lustre: Mounted lustre-client - version 2.17.57_1_g147bb23
[ 5456.665935] Lustre: Unmounted lustre-client
[ 5459.874535] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 21:35:56 (1787103356)
[ 5460.147479] LustreError: 87543:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5461.470510] LustreError: lustre-OST0000-osc-ffffa0eb5192f000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5461.483499] LustreError: 87631:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffffa0eb5192f000: namespace resource [0x240000400:0x544d:0x0].0x0 (ffffa0eb46c35a00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5461.490274] LustreError: 87631:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5461.608075] LustreError: 87543:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5461.611601] Lustre: 2359:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.139@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5464.675631] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 21:36:01 (1787103361)
[ 5525.367839] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 21:37:02 (1787103422)
[ 5525.923318] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 5526.587518] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 21:37:03 (1787103423)
[ 5582.816329] Lustre: lustre-MDT0000-mdc-ffffa0eb5192f000: Connection to lustre-MDT0000 (at 192.168.202.139@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5582.822051] Lustre: Skipped 17 previous similar messages
[ 5585.960153] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 5586.673687] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 21:38:03 (1787103483)
[ 5644.988657] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 21:39:01 (1787103541)
[ 5646.119388] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 5646.914181] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 21:39:03 (1787103543)
[ 5647.667883] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 5648.430823] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 21:39:05 (1787103545)
[ 5661.752398] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 21:39:18 (1787103558)
[ 5665.969862] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5685.219040] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 192.168.202.139@tcp) was lost; in progress operations using this service will fail
[ 5685.223227] LustreError: Skipped 2 previous similar messages
[ 5685.228241] Lustre: Evicted from MGS (at 192.168.202.139@tcp) after server handle changed from 0x5e599baba9ff1c8c to 0x5e599baba9ff37cf
[ 5685.231549] Lustre: Skipped 2 previous similar messages
[ 5686.990381] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5687.575185] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5691.757110] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 21:39:48 (1787103588)
[ 5692.564366] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 5693.106409] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 21:39:50 (1787103590)
[ 5701.683861] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 21:39:58 (1787103598)
[ 5710.824980] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 192.168.202.139@tcp) was lost; in progress operations using this service will fail
[ 5710.830029] LustreError: Skipped 1 previous similar message
[ 5710.834493] Lustre: Evicted from MGS (at 192.168.202.139@tcp) after server handle changed from 0x5e599baba9ff39f1 to 0x5e599baba9ff3cfa
[ 5710.837917] Lustre: Skipped 1 previous similar message
[ 5716.688846] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 21:40:13 (1787103613)
[ 5741.782037] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5743.306202] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5822.496469] LustreError: lustre-MDT0000-mdc-ffffa0eb5192f000: operation mds_statfs to node 192.168.202.139@tcp failed: rc = -107
[ 5822.499995] LustreError: Skipped 13 previous similar messages
[ 5823.456270] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 192.168.202.139@tcp) was lost; in progress operations using this service will fail
[ 5823.467433] Lustre: Evicted from MGS (at 192.168.202.139@tcp) after server handle changed from 0x5e599baba9ff3cfa to 0x5e599baba9ff4e03
[ 5824.465563] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5824.967833] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5845.968094] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5846.559672] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5863.392127] Lustre: 2360:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787103705/real 1787103705]  req@ffffa0eb491c5c00 x1873908119948160/t0(0) o400->lustre-MDT0000-mdc-ffffa0eb5192f000@192.168.202.139@tcp:12/10 lens 224/224 e 0 to 1 dl 1787103760 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5863.401656] Lustre: 2360:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 10 previous similar messages
[ 5908.484598] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 21:43:25 (1787103805)
[ 5935.093890] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5936.372670] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6109.775275] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 21:46:46 (1787104006)
[ 6212.586509] Lustre: lustre-MDT0000-mdc-ffffa0eb5192f000: Connection to lustre-MDT0000 (at 192.168.202.139@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6212.602309] Lustre: Skipped 8 previous similar messages
[ 6212.605160] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 192.168.202.139@tcp) was lost; in progress operations using this service will fail
[ 6212.613293] LustreError: Skipped 1 previous similar message
[ 6212.620303] Lustre: Evicted from MGS (at 192.168.202.139@tcp) after server handle changed from 0x5e599baba9ff4fd1 to 0x5e599babaa0d7000
[ 6212.625434] Lustre: Skipped 1 previous similar message
[ 6212.631860] Lustre: MGC192.168.202.139@tcp: Connection restored to 192.168.202.139@tcp (at 192.168.202.139@tcp)
[ 6212.638091] Lustre: Skipped 15 previous similar messages
[ 6233.608659] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 21:48:50 (1787104130)
[ 6234.424922] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 6235.321796] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 21:48:52 (1787104132)
[ 6236.420096] LustreError: lustre-MDT0000-mdc-ffffa0eb5192f000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6240.518864] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 21:48:57 (1787104137)
[ 6406.636649] Lustre: Evicted from lustre-OST0000_UUID (at 192.168.202.139@tcp) after server handle changed from 0x5e599babaa0d770e to 0x5e599babaa0d7818
[ 6406.642384] LustreError: lustre-OST0000-osc-ffffa0eb5192f000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6407.636903] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 21:51:44 (1787104304)
[ 6444.435899] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 21:52:21 (1787104341)
[ 6445.305374] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 6446.261112] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 21:52:23 (1787104343)
[ 6447.067515] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 6447.956374] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 21:52:24 (1787104344)
[ 6474.580664] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 21:52:51 (1787104371)
[ 6509.026494] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 192.168.202.139@tcp) was lost; in progress operations using this service will fail
[ 6512.340719] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 21:53:29 (1787104409)
[ 6513.031379] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 6513.757486] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 21:53:30 (1787104410)
[ 6514.447521] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 6515.325204] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 21:53:32 (1787104412)
[ 6521.492140] Lustre: Unmounted lustre-client
[ 6522.094421] Lustre: Mounted lustre-client - version 2.17.57_1_g147bb23
[ 6523.396277] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6541.343502] LustreError: lustre-MDT0000-mdc-ffffa0eb48139000: operation ldlm_enqueue to node 192.168.202.139@tcp failed: rc = -107
[ 6541.348568] LustreError: Skipped 8 previous similar messages
[ 6542.845640] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2