[ 2128.715728] LustreError: lustre-OST0000-osc-ffff9bf85122d800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2128.734173] Lustre: 2402:0:(llite_lib.c:4366:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.103@tcp:/lustre/fid: [0x200000405:0x2e:0x0]// may get corrupted (rc -5)
[ 2128.753490] Lustre: 2401:0:(llite_lib.c:4366:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.103@tcp:/lustre/fid: [0x200000405:0x2f:0x0]// may get corrupted (rc -108)
[ 2138.278395] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 08:26:45 (1789129605)
[ 2141.411480] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 2143.992876] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 08:26:51 (1789129611)
[ 2146.648312] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 2149.102237] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 08:26:56 (1789129616)
[ 2152.747578] LustreError: lustre-MDT0000-mdc-ffff9bf85122d800: operation mds_reint to node 192.168.201.103@tcp failed: rc = -19
[ 2152.771072] LustreError: Skipped 1 previous similar message
[ 2170.783538] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 192.168.201.103@tcp) was lost; in progress operations using this service will fail
[ 2180.075309] Lustre: Evicted from MGS (at 192.168.201.103@tcp) after server handle changed from 0x1e9bda66617872d5 to 0x1e9bda6661788b8d
[ 2283.473784] LustreError: lustre-MDT0000-mdc-ffff9bf85122d800: operation mds_reint to node 192.168.201.103@tcp failed: rc = -19
[ 2283.477897] LustreError: Skipped 1 previous similar message
[ 2303.969399] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 192.168.201.103@tcp) was lost; in progress operations using this service will fail
[ 2314.245517] Lustre: Evicted from MGS (at 192.168.201.103@tcp) after server handle changed from 0x1e9bda6661788b8d to 0x1e9bda66617a2ec2
[ 2330.509374] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 08:29:58 (1789129798)
[ 2331.287370] Lustre: *** cfs_fail_loc=305, val=0***
[ 2331.290447] Lustre: Skipped 2 previous similar messages
[ 2355.204841] Lustre: lustre-MDT0000-mdc-ffff9bf85122d800: Connection to lustre-MDT0000 (at 192.168.201.103@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2355.219779] Lustre: Skipped 11 previous similar messages
[ 2371.551372] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 192.168.201.103@tcp) was lost; in progress operations using this service will fail
[ 2381.811693] Lustre: Evicted from MGS (at 192.168.201.103@tcp) after server handle changed from 0x1e9bda66617a2ec2 to 0x1e9bda66617a430c
[ 2381.830246] Lustre: MGC192.168.201.103@tcp: Connection restored to 192.168.201.103@tcp (at 192.168.201.103@tcp)
[ 2381.838666] Lustre: Skipped 13 previous similar messages
[ 2383.023658] Lustre: 17119:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.103@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2405.746868] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2408.436456] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2422.264291] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 08:31:29 (1789129889)
[ 2443.252580] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 192.168.201.103@tcp) was lost; in progress operations using this service will fail
[ 2443.295766] Lustre: Evicted from MGS (at 192.168.201.103@tcp) after server handle changed from 0x1e9bda66617a430c to 0x1e9bda66617a4933
[ 2445.160747] LustreError: lustre-MDT0000-mdc-ffff9bf85122d800: operation ldlm_enqueue to node 192.168.201.103@tcp failed: rc = -107
[ 2448.380244] LustreError: lustre-MDT0000-mdc-ffff9bf85122d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2485.838907] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 08:32:33 (1789129953)
[ 2504.693671] LustreError: lustre-OST0000-osc-ffff9bf85122d800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2535.247595] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 08:33:22 (1789130002)
[ 2572.640539] Lustre: 2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789130025/real 1789130025]  req@ffff9bf8437812c0 x1876036217312128/t0(0) o400->MGC192.168.201.103@tcp@192.168.201.103@tcp:26/25 lens 224/224 e 0 to 1 dl 1789130041 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2572.681336] Lustre: 2400:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 9 previous similar messages
[ 2572.689812] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 192.168.201.103@tcp) was lost; in progress operations using this service will fail
[ 2582.040620] Lustre: Evicted from MGS (at 192.168.201.103@tcp) after server handle changed from 0x1e9bda66617a4933 to 0x1e9bda66617b0f1d
[ 2602.692918] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2606.356394] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2694.894807] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 192.168.201.103@tcp) was lost; in progress operations using this service will fail
[ 2705.410918] Lustre: Evicted from MGS (at 192.168.201.103@tcp) after server handle changed from 0x1e9bda66617b0f1d to 0x1e9bda66617d26f1
[ 2731.416114] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2734.245554] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2820.319168] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 192.168.201.103@tcp) was lost; in progress operations using this service will fail
[ 2830.910537] Lustre: Evicted from MGS (at 192.168.201.103@tcp) after server handle changed from 0x1e9bda66617d26f1 to 0x1e9bda66617f8941
[ 2853.263714] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2855.538941] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2884.086272] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 08:39:12 (1789130352)
[ 2888.461402] LustreError: lustre-MDT0000-mdc-ffff9bf85122d800: operation mds_reint to node 192.168.201.103@tcp failed: rc = -19
[ 2888.470429] LustreError: Skipped 1 previous similar message
[ 2907.615321] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 192.168.201.103@tcp) was lost; in progress operations using this service will fail
[ 2917.890148] Lustre: Evicted from MGS (at 192.168.201.103@tcp) after server handle changed from 0x1e9bda66617f8941 to 0x1e9bda6661808078
[ 2927.626303] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2971.400942] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2979.369391] Lustre: lustre-MDT0000-mdc-ffff9bf85122d800: Connection to lustre-MDT0000 (at 192.168.201.103@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2979.381371] Lustre: Skipped 7 previous similar messages
[ 3006.462483] Lustre: MGC192.168.201.103@tcp: Connection restored to 192.168.201.103@tcp (at 192.168.201.103@tcp)
[ 3006.479955] Lustre: Skipped 13 previous similar messages
[ 3021.629061] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 3056.607934] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 192.168.201.103@tcp) was lost; in progress operations using this service will fail
[ 3056.616800] LustreError: Skipped 2 previous similar messages
[ 3066.870076] Lustre: Evicted from MGS (at 192.168.201.103@tcp) after server handle changed from 0x1e9bda666180fb4c to 0x1e9bda6661816b5a
[ 3066.882062] Lustre: Skipped 2 previous similar messages
[ 3075.948068] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 3139.215757] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 3168.459192] LustreError: lustre-MDT0000-mdc-ffff9bf85122d800: operation mds_close to node 192.168.201.103@tcp failed: rc = -19
[ 3168.479235] LustreError: Skipped 7 previous similar messages
[ 3187.174571] Lustre: 2399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789130640/real 1789130640]  req@ffff9bf843621680 x1876036223867392/t0(0) o400->MGC192.168.201.103@tcp@192.168.201.103@tcp:26/25 lens 224/224 e 0 to 1 dl 1789130656 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3187.218550] Lustre: 2399:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
[ 3206.993241] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 3308.883740] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 08:46:16 (1789130776)
[ 3370.548834] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3372.842405] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3656.351318] Lustre: lustre-OST0000-osc-ffff9bf85122d800: Connection to lustre-OST0000 (at 192.168.201.103@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3656.365357] Lustre: Skipped 5 previous similar messages
[ 3678.887035] Lustre: lustre-OST0000-osc-ffff9bf85122d800: Connection restored to 192.168.201.103@tcp (at 192.168.201.103@tcp)
[ 3678.917061] Lustre: Skipped 10 previous similar messages
[ 3700.689471] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3703.424980] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3989.010602] LustreError: lustre-OST0000-osc-ffff9bf85122d800: operation ldlm_enqueue to node 192.168.201.103@tcp failed: rc = -107
[ 3989.033821] LustreError: Skipped 4 previous similar messages
[ 4033.378705] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4036.596884] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4284.791160] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 09:02:32 (1789131752)
[ 4303.839337] Lustre: 52249:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789131756/real 1789131756]  req@ffff9bf843494f00 x1876036241835392/t0(0) o101->lustre-MDT0000-mdc-ffff9bf85122d800@192.168.201.103@tcp:12/10 lens 576/1152 e 0 to 1 dl 1789131772 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 4303.889119] Lustre: 52249:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 4303.898468] Lustre: lustre-MDT0000-mdc-ffff9bf85122d800: Connection to lustre-MDT0000 (at 192.168.201.103@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4303.924474] Lustre: Skipped 1 previous similar message
[ 4304.009369] Lustre: lustre-MDT0000-mdc-ffff9bf85122d800: Connection restored to 192.168.201.103@tcp (at 192.168.201.103@tcp)
[ 4304.023359] Lustre: Skipped 1 previous similar message
[ 4314.849787] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 09:03:02 (1789131782)
[ 4341.288503] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 09:03:28 (1789131808)
[ 4370.777486] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 09:03:58 (1789131838)
[ 4371.137582] Lustre: Mounted lustre-client - version 2.17.57_135_g8edbab4
[ 4403.168787] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 192.168.201.103@tcp) was lost; in progress operations using this service will fail
[ 4403.187497] LustreError: Skipped 3 previous similar messages
[ 4413.431045] Lustre: Evicted from MGS (at 192.168.201.103@tcp) after server handle changed from 0x1e9bda66618385b9 to 0x1e9bda666198b9fa
[ 4413.449298] Lustre: Skipped 3 previous similar messages
[ 4431.773907] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4434.074636] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4436.407696] Lustre: Unmounted lustre-client
[ 4445.330765] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 09:05:12 (1789131912)
[ 4551.536651] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 09:06:58 (1789132018)
[ 4601.136468] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 09:07:48 (1789132068)
[ 4605.677113] Lustre: Unmounted lustre-client
[ 4644.230567] Lustre: Mounted lustre-client - version 2.17.57_135_g8edbab4
[ 4652.714563] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 09:08:40 (1789132120)
[ 4652.922910] LustreError: 58133:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4653.970560] LustreError: 58133:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4654.214087] Lustre: *** cfs_fail_loc=305, val=0***
[ 4678.831659] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 09:09:05 (1789132145)
[ 4679.515783] Lustre: Mounted lustre-client - version 2.17.57_135_g8edbab4
[ 4682.454285] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4692.807474] Lustre: Unmounted lustre-client
[ 4702.442372] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 09:09:29 (1789132169)
[ 4787.798554] LustreError: lustre-MDT0000-mdc-ffff9bf8440b6800: operation mds_reint to node 192.168.201.103@tcp failed: rc = -19
[ 4809.696448] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 192.168.201.103@tcp) was lost; in progress operations using this service will fail
[ 4809.717908] Lustre: Evicted from MGS (at 192.168.201.103@tcp) after server handle changed from 0x1e9bda666198cd72 to 0x1e9bda66619b7e66
[ 4811.638826] Lustre: 57520:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.103@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 4941.773090] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 09:13:29 (1789132409)
[ 4951.984762] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4960.235087] Lustre: lustre-MDT0000-mdc-ffff9bf8440b6800: Connection to lustre-MDT0000 (at 192.168.201.103@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4960.248205] Lustre: Skipped 12 previous similar messages
[ 4975.601990] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 192.168.201.103@tcp) was lost; in progress operations using this service will fail
[ 4975.605109] LustreError: lustre-MDT0000-mdc-ffff9bf8440b6800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4975.642593] Lustre: Evicted from MGS (at 192.168.201.103@tcp) after server handle changed from 0x1e9bda66619b7e66 to 0x1e9bda66619f4110
[ 4975.653511] Lustre: lustre-MDT0000-mdc-ffff9bf8440b6800: Connection restored to 192.168.201.103@tcp (at 192.168.201.103@tcp)
[ 4975.668828] Lustre: Skipped 15 previous similar messages
[ 5001.076781] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 09:14:28 (1789132468)
[ 5001.689637] Lustre: Mounted lustre-client - version 2.17.57_135_g8edbab4
[ 5009.085879] LustreError: lustre-OST0000-osc-ffff9bf8440b6800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5009.097309] LustreError: 62153:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff9bf8440b6800: namespace resource [0x280000401:0x3323:0x0].0x0 (ffff9bf847067200) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5019.103345] Lustre: 2401:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789132472/real 1789132472]  req@ffff9bf846901e00 x1876036246192128/t0(0) o103->lustre-OST0000-osc-ffff9bf8581bd000@192.168.201.103@tcp:17/18 lens 328/224 e 0 to 1 dl 1789132488 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295
[ 5019.162868] Lustre: 2401:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 41 previous similar messages
[ 5022.002452] Lustre: Unmounted lustre-client
[ 5031.609550] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 09:14:58 (1789132498)
[ 5038.982594] LustreError: lustre-MDT0000-mdc-ffff9bf8440b6800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5039.009372] LustreError: 62924:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9bf8440b6800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff9bf846bd1500) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5039.056682] LustreError: 62841:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 5048.824390] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 09:15:15 (1789132515)
[ 5049.030341] LustreError: 63549:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 5053.340929] LustreError: lustre-MDT0000-mdc-ffff9bf8440b6800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5053.364090] LustreError: 63570:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 5053.378481] LustreError: 63549:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=658
[ 5053.381703] LustreError: 63549:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting
[ 5053.431891] LustreError: 63575:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5054.569039] LustreError: 63581:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5054.582923] LustreError: 63581:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5056.908727] LustreError: 63604:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5056.932618] LustreError: 63604:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5061.914335] LustreError: 63649:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5061.938555] LustreError: 63649:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 5073.173700] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 09:15:40 (1789132540)
[ 5113.963035] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5116.222514] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5126.540247] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 09:16:34 (1789132594)
[ 5168.923419] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5170.706084] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5182.318740] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 09:17:29 (1789132649)
[ 5258.194897] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5260.412188] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5271.586522] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 09:18:58 (1789132738)
[ 5314.215735] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5315.974190] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5323.162260] Lustre: Unmounted lustre-client
[ 5383.002171] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5386.350377] Lustre: Mounted lustre-client - version 2.17.57_135_g8edbab4
[ 5395.176084] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 09:21:02 (1789132862)
[ 5398.004163] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5400.032080] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 09:21:07 (1789132867)
[ 5439.106630] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 09:21:46 (1789132906)
[ 5440.689739] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5442.539232] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 09:21:50 (1789132910)
[ 5444.060521] Lustre: *** cfs_fail_loc=805, val=0***
[ 5444.146082] Lustre: Mounted lustre-client - version 2.17.57_135_g8edbab4
[ 5455.381669] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5475.807326] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 192.168.201.103@tcp) was lost; in progress operations using this service will fail
[ 5500.394631] Lustre: Evicted from MGS (at 192.168.201.103@tcp) after server handle changed from 0x1e9bda66619f4c62 to 0x1e9bda66619f4eb5
[ 5501.546775] Lustre: 70243:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.103@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 5513.560947] Lustre: Unmounted lustre-client
[ 5522.607273] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 09:23:10 (1789132990)
[ 5555.740712] LustreError: 2398:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff9bf849208780 x1876036246299008/t94489280516(94489280516) o101->lustre-MDT0000-mdc-ffff9bf8581bf000@192.168.201.103@tcp:12/10 lens 520/664 e 0 to 0 dl 1789133040 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5568.897172] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5570.771508] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5580.752680] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 09:24:08 (1789133048)
[ 5586.408699] Lustre: lustre-OST0000-osc-ffff9bf8581bf000: Connection to lustre-OST0000 (at 192.168.201.103@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5586.416352] Lustre: Skipped 13 previous similar messages
[ 5586.449527] LustreError: 2398:0:(osc_request.c:1026:osc_init_grant()) lustre-OST0000-osc-ffff9bf8581bf000: granted 8437760 but already consumed 12632064
[ 5586.461783] LustreError: lustre-OST0000-osc-ffff9bf8581bf000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5586.473928] Lustre: 2402:0:(llite_lib.c:4366:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.103@tcp:/lustre/fid: [0x240000408:0x2:0x0]// may get corrupted (rc -5)
[ 5586.639296] LustreError: 74888:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff9bf8581bf000: namespace resource [0x280000400:0xf56:0x0].0x0 (ffff9bf846bd1500) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5595.528642] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 09:24:23 (1789133063)
[ 5614.092741] Lustre: lustre-MDT0000-mdc-ffff9bf8581bf000: Connection restored to 192.168.201.103@tcp (at 192.168.201.103@tcp)
[ 5614.105966] Lustre: Skipped 13 previous similar messages
[ 5622.650288] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 09:24:50 (1789133090)
[ 5640.159328] Lustre: 76151:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789133093/real 1789133093]  req@ffff9bf846902940 x1876036246326016/t0(0) o36->lustre-MDT0001-mdc-ffff9bf8581bf000@192.168.201.103@tcp:12/10 lens 560/576 e 0 to 1 dl 1789133109 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295
[ 5640.218573] Lustre: 76151:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
[ 5649.863948] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 09:25:17 (1789133117)
[ 5678.255509] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 09:25:45 (1789133145)
[ 5746.661270] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 09:26:54 (1789133214)
[ 5815.814566] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 09:28:02 (1789133282)
[ 5841.079585] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 09:28:28 (1789133308)
[ 5905.712355] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 09:29:33 (1789133373)
[ 5931.350919] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 09:29:58 (1789133398)
[ 5957.081793] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 09:30:24 (1789133424)
[ 5983.086749] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 09:30:50 (1789133450)
[ 5983.377422] Lustre: Unmounted lustre-client
[ 6029.429172] Lustre: Mounted lustre-client - version 2.17.57_135_g8edbab4
[ 6046.056452] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 09:31:54 (1789133514)
[ 6064.319442] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 09:32:12 (1789133532)
[ 6086.112913] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 192.168.201.103@tcp) was lost; in progress operations using this service will fail
[ 6086.135186] LustreError: Skipped 1 previous similar message
[ 6096.406857] Lustre: Evicted from MGS (at 192.168.201.103@tcp) after server handle changed from 0x1e9bda66619f7439 to 0x1e9bda66619f7b6a
[ 6096.427242] Lustre: Skipped 1 previous similar message
[ 6101.522194] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 09:32:48 (1789133568)
[ 6135.413793] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 09:33:22 (1789133602)
[ 6136.321995] Lustre: *** cfs_fail_loc=51b, val=3***
[ 6152.258720] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 09:33:38 (1789133618)
[ 6153.111852] Lustre: *** cfs_fail_loc=51b, val=4***
[ 6160.676986] Lustre: DEBUG MARKER: recovery-small test_115b: @@@@@@ FAIL: dd success