[ 1923.320324] LustreError: lustre-OST0000-osc-ffff8b8d90c80000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1923.332038] Lustre: 2364:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.206.137@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1928.557835] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 22:16:29 (1788747389)
[ 1930.263266] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1931.662082] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 22:16:32 (1788747392)
[ 1933.154832] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1934.393663] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 22:16:35 (1788747395)
[ 1937.397152] LustreError: lustre-MDT0000-mdc-ffff8b8d90c80000: operation mds_close to node 192.168.206.137@tcp failed: rc = -19
[ 1937.404312] LustreError: Skipped 2 previous similar messages
[ 1955.817554] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 192.168.206.137@tcp) was lost; in progress operations using this service will fail
[ 1955.837383] Lustre: Evicted from MGS (at 192.168.206.137@tcp) after server handle changed from 0xd1e627d98cba8d12 to 0xd1e627d98cbaa81d
[ 1957.167737] Lustre: 15932:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.137@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2063.103087] LustreError: lustre-MDT0000-mdc-ffff8b8d90c80000: operation mds_reint to node 192.168.206.137@tcp failed: rc = -19
[ 2063.116144] LustreError: Skipped 1 previous similar message
[ 2082.799356] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 192.168.206.137@tcp) was lost; in progress operations using this service will fail
[ 2082.872590] Lustre: Evicted from MGS (at 192.168.206.137@tcp) after server handle changed from 0xd1e627d98cbaa81d to 0xd1e627d98cbd2173
[ 2090.235400] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 22:19:11 (1788747551)
[ 2090.528892] Lustre: *** cfs_fail_loc=305, val=0***
[ 2090.533533] Lustre: Skipped 2 previous similar messages
[ 2128.863789] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 192.168.206.137@tcp) was lost; in progress operations using this service will fail
[ 2128.888741] Lustre: Evicted from MGS (at 192.168.206.137@tcp) after server handle changed from 0xd1e627d98cbd2173 to 0xd1e627d98cbd3e92
[ 2138.213207] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2139.294855] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2144.833716] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 22:20:06 (1788747606)
[ 2154.473947] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 192.168.206.137@tcp) was lost; in progress operations using this service will fail
[ 2154.496936] Lustre: Evicted from MGS (at 192.168.206.137@tcp) after server handle changed from 0xd1e627d98cbd3e92 to 0xd1e627d98cbd4315
[ 2154.523869] LustreError: lustre-MDT0000-mdc-ffff8b8d90c80000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2168.425934] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 22:20:29 (1788747629)
[ 2180.070558] LustreError: lustre-OST0000-osc-ffff8b8d90c80000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2193.404517] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 22:20:54 (1788747654)
[ 2205.802952] LustreError: lustre-MDT0000-mdc-ffff8b8d90c80000: operation mds_reint to node 192.168.206.137@tcp failed: rc = -19
[ 2205.810063] LustreError: Skipped 1 previous similar message
[ 2226.159174] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 192.168.206.137@tcp) was lost; in progress operations using this service will fail
[ 2226.231275] Lustre: Evicted from MGS (at 192.168.206.137@tcp) after server handle changed from 0xd1e627d98cbd4315 to 0xd1e627d98cbd9b21
[ 2236.713676] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2238.134671] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2302.597290] Lustre: lustre-MDT0000-mdc-ffff8b8d90c80000: Connection to lustre-MDT0000 (at 192.168.206.137@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2302.618861] Lustre: Skipped 14 previous similar messages
[ 2318.815498] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 192.168.206.137@tcp) was lost; in progress operations using this service will fail
[ 2323.029039] Lustre: lustre-MDT0000-mdc-ffff8b8d90c80000: Connection restored to 192.168.206.137@tcp (at 192.168.206.137@tcp)
[ 2323.032678] Lustre: Skipped 19 previous similar messages
[ 2329.127187] Lustre: Evicted from MGS (at 192.168.206.137@tcp) after server handle changed from 0xd1e627d98cbd9b21 to 0xd1e627d98cc01af9
[ 2331.289939] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2333.148232] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2397.808520] LustreError: lustre-MDT0000-mdc-ffff8b8d90c80000: operation ldlm_enqueue to node 192.168.206.137@tcp failed: rc = -19
[ 2397.813530] LustreError: Skipped 1 previous similar message
[ 2416.110996] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 192.168.206.137@tcp) was lost; in progress operations using this service will fail
[ 2416.128915] Lustre: Evicted from MGS (at 192.168.206.137@tcp) after server handle changed from 0xd1e627d98cc01af9 to 0xd1e627d98cc1f0bd
[ 2431.637119] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2433.416188] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2462.445808] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 22:25:23 (1788747923)
[ 2482.719139] Lustre: 2363:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788747929/real 1788747929]  req@ffff8b8d83771500 x1875635730554112/t0(0) o400->MGC192.168.206.137@tcp@192.168.206.137@tcp:26/25 lens 224/224 e 0 to 1 dl 1788747945 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2482.736332] Lustre: 2363:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 10 previous similar messages
[ 2482.745980] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 192.168.206.137@tcp) was lost; in progress operations using this service will fail
[ 2492.968956] Lustre: Evicted from MGS (at 192.168.206.137@tcp) after server handle changed from 0xd1e627d98cc1f0bd to 0xd1e627d98cc2f3bd
[ 2500.839134] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2528.571508] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2572.006060] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2611.225359] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2649.608865] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 192.168.206.137@tcp) was lost; in progress operations using this service will fail
[ 2649.619821] LustreError: Skipped 3 previous similar messages
[ 2659.823825] Lustre: Evicted from MGS (at 192.168.206.137@tcp) after server handle changed from 0xd1e627d98cc3b6cf to 0xd1e627d98cc4963c
[ 2659.831384] Lustre: Skipped 3 previous similar messages
[ 2666.533856] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2694.206376] LustreError: lustre-MDT0000-mdc-ffff8b8d90c80000: operation ldlm_enqueue to node 192.168.206.137@tcp failed: rc = -19
[ 2694.221085] LustreError: Skipped 7 previous similar messages
[ 2730.936202] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2815.430456] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 22:31:16 (1788748276)
[ 2863.715264] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 2865.760225] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3160.270304] Lustre: lustre-OST0000-osc-ffff8b8d90c80000: Connection to lustre-OST0000 (at 192.168.206.137@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3160.286536] Lustre: Skipped 9 previous similar messages
[ 3182.525121] Lustre: lustre-OST0000-osc-ffff8b8d90c80000: Connection restored to 192.168.206.137@tcp (at 192.168.206.137@tcp)
[ 3182.532817] Lustre: Skipped 18 previous similar messages
[ 3197.171889] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3199.329791] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3490.468782] LustreError: lustre-OST0000-osc-ffff8b8d90c80000: operation ldlm_enqueue to node 192.168.206.137@tcp failed: rc = -107
[ 3490.477540] LustreError: Skipped 5 previous similar messages
[ 3522.352056] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3523.905164] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3784.466535] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 22:47:26 (1788749246)
[ 3801.055160] Lustre: 50615:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788749247/real 1788749247]  req@ffff8b8d83702d80 x1875635756018560/t0(0) o101->lustre-MDT0000-mdc-ffff8b8d90c80000@192.168.206.137@tcp:12/10 lens 576/1152 e 0 to 1 dl 1788749263 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3801.066321] Lustre: 50615:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
[ 3801.068930] Lustre: lustre-MDT0000-mdc-ffff8b8d90c80000: Connection to lustre-MDT0000 (at 192.168.206.137@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3801.077555] Lustre: Skipped 1 previous similar message
[ 3801.099373] Lustre: lustre-MDT0000-mdc-ffff8b8d90c80000: Connection restored to 192.168.206.137@tcp (at 192.168.206.137@tcp)
[ 3801.103717] Lustre: Skipped 1 previous similar message
[ 3805.100213] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 22:47:46 (1788749266)
[ 3826.796809] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 22:48:08 (1788749288)
[ 3847.066445] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 22:48:28 (1788749308)
[ 3847.222735] Lustre: Mounted lustre-client - version 2.17.58_40_g869f601
[ 3872.742638] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 192.168.206.137@tcp) was lost; in progress operations using this service will fail
[ 3872.752577] LustreError: Skipped 2 previous similar messages
[ 3872.760345] Lustre: Evicted from MGS (at 192.168.206.137@tcp) after server handle changed from 0xd1e627d98cc60b65 to 0xd1e627d98ce0ee41
[ 3872.766330] Lustre: Skipped 2 previous similar messages
[ 3877.857238] Lustre: 2364:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788749324/real 1788749324]  req@ffff8b8d83697800 x1875635756038016/t0(0) o400->lustre-MDT0000-mdc-ffff8b8d837ff000@192.168.206.137@tcp:12/10 lens 224/224 e 0 to 1 dl 1788749340 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3877.876776] Lustre: 2364:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 3880.410857] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 3881.204362] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3882.753761] Lustre: Unmounted lustre-client
[ 3885.815572] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 22:49:07 (1788749347)
[ 4036.448192] Lustre: 2364:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788749482/real 1788749482]  req@ffff8b8d83694a80 x1875635756068096/t0(0) o4->lustre-OST0000-osc-ffff8b8d90c80000@192.168.206.137@tcp:6/4 lens 488/448 e 0 to 1 dl 1788749498 ref 2 fl Rpc:XQr/602/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4036.463358] Lustre: 2364:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 74 previous similar messages
[ 4171.721813] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 22:53:53 (1788749633)
[ 4215.738876] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 22:54:37 (1788749677)
[ 4217.390556] Lustre: Unmounted lustre-client
[ 4227.728726] Lustre: Mounted lustre-client - version 2.17.58_40_g869f601
[ 4230.677207] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 22:54:52 (1788749692)
[ 4230.775939] LustreError: 56525:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4231.815125] LustreError: 56525:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4231.854213] Lustre: *** cfs_fail_loc=305, val=0***
[ 4250.557135] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 22:55:12 (1788749712)
[ 4250.738741] Lustre: Mounted lustre-client - version 2.17.58_40_g869f601
[ 4251.311217] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4261.555796] Lustre: Unmounted lustre-client
[ 4265.002471] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 22:55:26 (1788749726)
[ 4315.617809] LustreError: lustre-MDT0000-mdc-ffff8b8d837e9800: operation mds_reint to node 192.168.206.137@tcp failed: rc = -19
[ 4333.034923] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 192.168.206.137@tcp) was lost; in progress operations using this service will fail
[ 4333.058078] Lustre: Evicted from MGS (at 192.168.206.137@tcp) after server handle changed from 0xd1e627d98ce10746 to 0xd1e627d98ce3d601
[ 4411.013530] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 22:57:52 (1788749872)
[ 4413.975265] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4424.420931] Lustre: lustre-MDT0000-mdc-ffff8b8d837e9800: Connection to lustre-MDT0000 (at 192.168.206.137@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4424.426439] Lustre: Skipped 24 previous similar messages
[ 4424.447227] LustreError: lustre-MDT0000-mdc-ffff8b8d837e9800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4424.461056] Lustre: lustre-MDT0000-mdc-ffff8b8d837e9800: Connection restored to 192.168.206.137@tcp (at 192.168.206.137@tcp)
[ 4424.465708] Lustre: Skipped 26 previous similar messages
[ 4425.187295] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 192.168.206.137@tcp) was lost; in progress operations using this service will fail
[ 4425.196074] Lustre: Evicted from MGS (at 192.168.206.137@tcp) after server handle changed from 0xd1e627d98ce3d601 to 0xd1e627d98ce872f4
[ 4428.534755] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 22:58:10 (1788749890)
[ 4428.720293] Lustre: Mounted lustre-client - version 2.17.58_40_g869f601
[ 4433.896320] LustreError: lustre-OST0000-osc-ffff8b8d837e9800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4433.901926] LustreError: 60215:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff8b8d837e9800: namespace resource [0x240000400:0x4cc3:0x0].0x0 (ffff8b8d87832100) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4445.152893] Lustre: 2361:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788749891/real 1788749891]  req@ffff8b8d841d7b80 x1875635760691712/t0(0) o103->lustre-OST0000-osc-ffff8b8d8338c000@192.168.206.137@tcp:17/18 lens 328/224 e 0 to 1 dl 1788749907 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295
[ 4445.170568] Lustre: 2361:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 73 previous similar messages
[ 4445.508169] Lustre: Unmounted lustre-client
[ 4447.719474] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 22:58:29 (1788749909)
[ 4452.716333] LustreError: lustre-MDT0000-mdc-ffff8b8d837e9800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4452.724971] LustreError: 60885:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4455.842469] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 22:58:37 (1788749917)
[ 4455.910051] LustreError: 61587:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4461.025548] LustreError: 61587:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=0
[ 4461.028962] LustreError: 61587:0:(import.c:756:ptlrpc_connect_import_locked()) already connected
[ 4461.949514] LustreError: lustre-MDT0000-mdc-ffff8b8d837e9800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4461.957971] LustreError: 61605:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4461.982439] LustreError: 61610:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4463.024876] LustreError: 61616:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4463.029472] LustreError: 61616:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4465.148150] LustreError: 61638:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4465.152579] LustreError: 61638:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4469.394875] LustreError: 61683:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4469.401211] LustreError: 61683:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4475.451171] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 22:58:57 (1788749937)
[ 4498.517742] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4499.157112] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4503.632135] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 22:59:25 (1788749965)
[ 4525.743788] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4526.397583] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4531.180265] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 22:59:53 (1788749993)
[ 4590.034977] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4592.482480] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4603.454434] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 23:01:04 (1788750064)
[ 4649.826603] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4651.853221] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4659.878497] Lustre: Unmounted lustre-client
[ 4703.887436] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4705.909486] Lustre: Mounted lustre-client - version 2.17.58_40_g869f601
[ 4710.278940] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 23:02:51 (1788750171)
[ 4712.130640] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 4713.410589] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 23:02:54 (1788750174)
[ 4738.202480] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 23:03:19 (1788750199)
[ 4739.409258] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 4740.971380] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 23:03:22 (1788750202)
[ 4742.252886] Lustre: *** cfs_fail_loc=805, val=0***
[ 4742.302339] Lustre: Mounted lustre-client - version 2.17.58_40_g869f601
[ 4746.991188] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4768.226403] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 192.168.206.137@tcp) was lost; in progress operations using this service will fail
[ 4768.243224] Lustre: Evicted from MGS (at 192.168.206.137@tcp) after server handle changed from 0xd1e627d98ce87cef to 0xd1e627d98ce88084
[ 4772.593917] Lustre: Unmounted lustre-client
[ 4778.484723] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 23:03:59 (1788750239)
[ 4805.066228] LustreError: 2360:0:(client.c:3449:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff8b8d89a25500 x1875635760768000/t94489280516(94489280516) o101->lustre-MDT0000-mdc-ffff8b8d8355a000@192.168.206.137@tcp:12/10 lens 664/608 e 0 to 0 dl 1788750283 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 4811.015476] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4812.677718] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4820.236769] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 23:04:41 (1788750281)
[ 4821.738286] LustreError: lustre-OST0000-osc-ffff8b8d8355a000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4821.758602] Lustre: 2361:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.206.137@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 4829.231645] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 23:04:50 (1788750290)
[ 4830.846739] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 4832.753249] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 23:04:53 (1788750293)
[ 4834.252388] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 4835.923548] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 23:04:57 (1788750297)
[ 4837.509383] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 4838.910613] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 23:05:00 (1788750300)
[ 4840.091597] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 4841.796493] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 23:05:02 (1788750302)
[ 4842.922970] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 4844.479424] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 23:05:05 (1788750305)
[ 4846.085359] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 4847.713650] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 23:05:08 (1788750308)
[ 4849.203897] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 4851.016549] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 23:05:12 (1788750312)
[ 4852.350804] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 4854.093642] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 23:05:15 (1788750315)
[ 4855.445865] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 4856.933138] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 23:05:18 (1788750318)
[ 4858.405658] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 4859.872148] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 23:05:21 (1788750321)
[ 4861.270868] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 4862.980595] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 23:05:24 (1788750324)
[ 4865.587626] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 4867.480709] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 23:05:28 (1788750328)
[ 4895.700266] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 23:05:56 (1788750356)
[ 4925.298365] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 23:06:26 (1788750386)
[ 4925.819427] Lustre: *** cfs_fail_loc=51b, val=3***
[ 4936.140428] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 23:06:37 (1788750397)
[ 4936.452918] Lustre: *** cfs_fail_loc=51b, val=4***
[ 4946.805967] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 23:06:48 (1788750408)
[ 4947.280240] Lustre: *** cfs_fail_loc=50f, val=3***
[ 4956.879445] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 23:06:57 (1788750417)
[ 4957.245610] Lustre: *** cfs_fail_loc=50f, val=4***
[ 4965.529247] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 23:07:06 (1788750426)
[ 4965.973358] Lustre: *** cfs_fail_loc=510, val=3***
[ 4974.565884] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 23:07:15 (1788750435)
[ 4975.329475] Lustre: *** cfs_fail_loc=51b, val=3***
[ 4986.531693] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 23:07:27 (1788750447)
[ 4987.204225] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5046.751160] Lustre: 2362:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788750453/real 1788750453]  req@ffff8b8d89a27100 x1875635760813056/t0(0) o3->lustre-OST0001-osc-ffff8b8d8355a000@192.168.206.137@tcp:6/4 lens 488/4536 e 0 to 1 dl 1788750508 ref 2 fl Rpc:RXQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 5046.778478] Lustre: 2362:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 13 previous similar messages
[ 5046.782876] Lustre: lustre-OST0001-osc-ffff8b8d8355a000: Connection to lustre-OST0001 (at 192.168.206.137@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5046.795975] Lustre: Skipped 16 previous similar messages
[ 5055.113815] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 23:08:35 (1788750515)
[ 5055.658516] LustreError: 83008:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5058.614437] LustreError: lustre-MDT0000-mdc-ffff8b8d8355a000: operation ldlm_enqueue to node 192.168.206.137@tcp failed: rc = -107
[ 5058.619614] LustreError: Skipped 4 previous similar messages
[ 5058.630779] LustreError: lustre-MDT0000-mdc-ffff8b8d8355a000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5058.657187] LustreError: 83023:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8b8d8355a000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8b8d85e0e300) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5058.667297] LustreError: 83023:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5058.674677] Lustre: lustre-MDT0000-mdc-ffff8b8d8355a000: Connection restored to 192.168.206.137@tcp (at 192.168.206.137@tcp)
[ 5058.687629] Lustre: Skipped 18 previous similar messages
[ 5059.751601] LustreError: 83008:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5061.910266] LustreError: 83039:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5065.257770] LustreError: lustre-MDT0000-mdc-ffff8b8d8355a000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5065.284089] LustreError: 83053:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8b8d8355a000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8b8d83619c00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5065.309540] LustreError: 83053:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5065.991115] LustreError: 83039:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5073.326574] LustreError: lustre-MDT0000-mdc-ffff8b8d8355a000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5073.813691] LustreError: 83094:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5076.984243] LustreError: lustre-MDT0000-mdc-ffff8b8d8355a000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5077.081516] LustreError: 83114:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5077.096286] LustreError: 83114:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 20 previous similar messages
[ 5077.887158] LustreError: 83094:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5077.891498] LustreError: 83094:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8b8d8355a000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5084.390794] LustreError: 83169:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5087.521471] LustreError: lustre-MDT0000-mdc-ffff8b8d8355a000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5087.548777] LustreError: 83184:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8b8d8355a000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8b8d85e0e700) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5087.573204] LustreError: 83184:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5088.471135] LustreError: 83169:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5095.637676] LustreError: lustre-MDT0000-mdc-ffff8b8d8355a000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5096.225371] LustreError: 83226:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5098.946514] LustreError: lustre-MDT0000-mdc-ffff8b8d8355a000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5098.987233] LustreError: 83240:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8b8d8355a000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8b8d85e0e100) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5099.023914] LustreError: 83240:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5100.295189] LustreError: 83226:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5103.365173] LustreError: lustre-MDT0000-mdc-ffff8b8d8355a000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5103.445918] LustreError: 83273:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5103.456476] LustreError: 83273:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5104.479674] LustreError: 83253:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8b8d8355a000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5112.546956] LustreError: 83325:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5112.561982] LustreError: 83325:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5115.917770] LustreError: lustre-MDT0000-mdc-ffff8b8d8355a000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5116.639182] LustreError: 83325:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5116.647272] LustreError: 83325:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5116.659659] LustreError: 83325:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8b8d8355a000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5126.546257] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 23:09:47 (1788750587)
[ 5145.891101] LustreError: 79760:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5145.901442] LustreError: 79760:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5149.975515] LustreError: 79760:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5149.986401] LustreError: 79760:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5158.073932] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 23:10:19 (1788750619)
[ 5210.415723] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 23:11:11 (1788750671)
[ 5277.258310] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 23:12:18 (1788750738)
[ 5312.644747] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 23:12:53 (1788750773)
[ 5313.441986] Lustre: Mounted lustre-client - version 2.17.58_40_g869f601
[ 5437.426199] Lustre: Unmounted lustre-client
[ 5444.762856] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 23:15:05 (1788750905)
[ 5445.395286] LustreError: 87650:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5447.927694] LustreError: lustre-OST0000-osc-ffff8b8d8355a000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5447.936453] LustreError: 87733:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff8b8d8355a000: namespace resource [0x240000400:0x4d4d:0x0].0x0 (ffff8b8d83619700) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5447.953508] LustreError: 87733:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5448.223210] LustreError: 87650:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5448.236580] Lustre: 2363:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.206.137@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5454.333502] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 23:15:15 (1788750915)
[ 5520.377920] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 23:16:21 (1788750981)
[ 5522.044842] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 5523.864678] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 23:16:24 (1788750984)
[ 5585.598482] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 5586.938360] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 23:17:28 (1788751048)
[ 5645.682503] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 23:18:26 (1788751106)
[ 5648.705790] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 5650.251540] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 23:18:31 (1788751111)
[ 5651.883437] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 5654.076569] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 23:18:34 (1788751114)
[ 5686.554411] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 23:19:07 (1788751147)
[ 5696.273455] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5721.062672] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 192.168.206.137@tcp) was lost; in progress operations using this service will fail
[ 5721.076870] LustreError: Skipped 2 previous similar messages
[ 5721.082403] Lustre: lustre-MDT0000-mdc-ffff8b8d8355a000: Connection to lustre-MDT0000 (at 192.168.206.137@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5721.098843] Lustre: Skipped 16 previous similar messages
[ 5721.099792] Lustre: Evicted from MGS (at 192.168.206.137@tcp) after server handle changed from 0xd1e627d98ce88b51 to 0xd1e627d98ce8a441
[ 5721.112840] Lustre: Skipped 2 previous similar messages
[ 5721.119648] Lustre: MGC192.168.206.137@tcp: Connection restored to 192.168.206.137@tcp (at 192.168.206.137@tcp)
[ 5721.128523] Lustre: Skipped 14 previous similar messages
[ 5728.099697] Lustre: 2361:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788751168/real 1788751168]  req@ffff8b8d89a25f80 x1875635760975872/t0(0) o400->lustre-MDT0000-mdc-ffff8b8d8355a000@192.168.206.137@tcp:12/10 lens 224/224 e 0 to 1 dl 1788751190 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5728.116335] Lustre: 2361:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
[ 5730.629873] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5731.862364] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5739.414110] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 23:20:00 (1788751200)
[ 5741.249659] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 5742.574177] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 23:20:03 (1788751203)
[ 5760.817953] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 23:20:22 (1788751222)
[ 5782.518738] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 192.168.206.137@tcp) was lost; in progress operations using this service will fail
[ 5782.527239] LustreError: Skipped 1 previous similar message
[ 5782.544692] Lustre: Evicted from MGS (at 192.168.206.137@tcp) after server handle changed from 0xd1e627d98ce8a7f9 to 0xd1e627d98ce8ab80
[ 5782.550854] Lustre: Skipped 1 previous similar message
[ 5796.206768] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 23:20:57 (1788751257)
[ 5801.909432] LustreError: lustre-OST0000-osc-ffff8b8d8355a000: operation ost_setattr to node 192.168.206.137@tcp failed: rc = -107
[ 5801.918739] LustreError: Skipped 9 previous similar messages
[ 5843.721356] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5845.672983] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5930.992093] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 192.168.206.137@tcp) was lost; in progress operations using this service will fail
[ 5931.009346] Lustre: Evicted from MGS (at 192.168.206.137@tcp) after server handle changed from 0xd1e627d98ce8ab80 to 0xd1e627d98ce8b853
[ 5932.850245] Lustre: 68265:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.137@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 5956.199565] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5957.504518] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5986.925582] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5988.020407] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6086.846534] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 23:25:47 (1788751547)
[ 6133.273156] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6135.204466] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6366.680487] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 23:30:27 (1788751827)
[ 6515.183180] Lustre: lustre-MDT0000-mdc-ffff8b8d8355a000: Connection to lustre-MDT0000 (at 192.168.206.137@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6515.204107] Lustre: Skipped 6 previous similar messages
[ 6515.217178] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 192.168.206.137@tcp) was lost; in progress operations using this service will fail
[ 6515.236118] LustreError: Skipped 1 previous similar message
[ 6515.252704] Lustre: Evicted from MGS (at 192.168.206.137@tcp) after server handle changed from 0xd1e627d98ce8bd62 to 0xd1e627d98cf86058
[ 6515.264519] Lustre: Skipped 1 previous similar message
[ 6515.283820] Lustre: MGC192.168.206.137@tcp: Connection restored to 192.168.206.137@tcp (at 192.168.206.137@tcp)
[ 6515.295184] Lustre: Skipped 9 previous similar messages
[ 6544.602273] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 23:33:25 (1788752005)
[ 6546.023325] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 6547.805881] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 23:33:28 (1788752008)
[ 6549.614232] LustreError: lustre-MDT0000-mdc-ffff8b8d8355a000: operation ldlm_enqueue to node 192.168.206.137@tcp failed: rc = -107
[ 6549.626729] LustreError: Skipped 173 previous similar messages
[ 6549.652704] LustreError: lustre-MDT0000-mdc-ffff8b8d8355a000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6556.517446] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 23:33:37 (1788752017)
[ 6723.564307] LustreError: lustre-OST0000-osc-ffff8b8d8355a000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6724.749114] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 23:36:26 (1788752186)
[ 6749.151148] Lustre: 2361:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788752191/real 1788752191]  req@ffff8b8db4c4c000 x1875635778653952/t0(0) o4->lustre-OST0000-osc-ffff8b8d8355a000@192.168.206.137@tcp:6/4 lens 4584/448 e 0 to 1 dl 1788752211 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 6749.176828] Lustre: 2361:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 9 previous similar messages
[ 6763.524794] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 23:37:05 (1788752225)
[ 6764.506377] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 6765.586687] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 23:37:07 (1788752227)
[ 6766.937618] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 6768.008080] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 23:37:09 (1788752229)
[ 6797.283875] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 23:37:38 (1788752258)
[ 6831.075896] LustreError: MGC192.168.206.137@tcp: Connection to MGS (at 192.168.206.137@tcp) was lost; in progress operations using this service will fail
[ 6831.104391] Lustre: Evicted from MGS (at 192.168.206.137@tcp) after server handle changed from 0xd1e627d98cf86058 to 0xd1e627d98cf889d3
[ 6831.109214] Lustre: Skipped 1 previous similar message
[ 6840.433961] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 23:38:21 (1788752301)
[ 6841.523831] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 6842.835493] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 23:38:24 (1788752304)
[ 6843.829121] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 6845.109422] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 23:38:26 (1788752306)
[ 6851.955580] Lustre: Unmounted lustre-client
[ 6852.928691] Lustre: Mounted lustre-client - version 2.17.58_40_g869f601
[ 6854.826602] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6880.339476] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2