[ 1896.962877] LustreError: 34761:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 1901.561458] LustreError: lustre-OST0000-osc-ffffa06f4a65b800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1907.008132] LustreError: 34761:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 awake
[ 1907.023477] Lustre: 2369:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.148@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1907.027651] LustreError: 34761:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 1917.064336] LustreError: 34761:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 awake
[ 1917.073361] Lustre: 2369:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.148@tcp:/lustre/fid: [0x200000404:0x36:0x0]// may get corrupted (rc -108)
[ 1924.091790] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 02:02:52 (1790920972)
[ 1926.727436] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1929.161359] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 02:02:57 (1790920977)
[ 1931.845426] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1933.637556] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 02:03:02 (1790920982)
[ 1937.276868] LustreError: lustre-MDT0000-mdc-ffffa06f4a65b800: operation mds_reint to node 192.168.204.148@tcp failed: rc = -19
[ 1957.781696] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 192.168.204.148@tcp) was lost; in progress operations using this service will fail
[ 1957.851931] Lustre: Evicted from MGS (at 192.168.204.148@tcp) after server handle changed from 0x5ca00a437326fc85 to 0x5ca00a4373271727
[ 2061.351043] LustreError: lustre-MDT0000-mdc-ffffa06f4a65b800: operation ldlm_enqueue to node 192.168.204.148@tcp failed: rc = -19
[ 2079.715457] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 192.168.204.148@tcp) was lost; in progress operations using this service will fail
[ 2089.970699] Lustre: Evicted from MGS (at 192.168.204.148@tcp) after server handle changed from 0x5ca00a4373271727 to 0x5ca00a437328b205
[ 2107.899853] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 02:05:56 (1790921156)
[ 2108.525429] Lustre: *** cfs_fail_loc=305, val=0***
[ 2108.532951] Lustre: Skipped 2 previous similar messages
[ 2129.042754] LustreError: lustre-OST0001-osc-ffffa06f4a65b800: operation ost_connect to node 192.168.204.148@tcp failed: rc = -75
[ 2156.000269] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 192.168.204.148@tcp) was lost; in progress operations using this service will fail
[ 2166.280764] Lustre: Evicted from MGS (at 192.168.204.148@tcp) after server handle changed from 0x5ca00a437328b205 to 0x5ca00a437328b70d
[ 2166.282337] Lustre: lustre-MDT0000-mdc-ffffa06f4a65b800: Connection to lustre-MDT0000 (at 192.168.204.148@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2166.306402] Lustre: MGC192.168.204.148@tcp: Connection restored to 192.168.204.148@tcp (at 192.168.204.148@tcp)
[ 2166.321573] Lustre: Skipped 11 previous similar messages
[ 2166.349018] Lustre: Skipped 13 previous similar messages
[ 2180.685595] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2182.685367] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2191.498380] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 02:07:20 (1790921240)
[ 2207.207973] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 192.168.204.148@tcp) was lost; in progress operations using this service will fail
[ 2207.239810] Lustre: Evicted from MGS (at 192.168.204.148@tcp) after server handle changed from 0x5ca00a437328b70d to 0x5ca00a437328bc5b
[ 2207.302616] LustreError: lustre-MDT0000-mdc-ffffa06f4a65b800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2231.581199] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 02:08:00 (1790921280)
[ 2248.184969] LustreError: lustre-OST0000-osc-ffffa06f4a65b800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2264.841551] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 02:08:34 (1790921314)
[ 2280.264055] LustreError: lustre-MDT0000-mdc-ffffa06f4a65b800: operation mds_close to node 192.168.204.148@tcp failed: rc = -19
[ 2280.282085] LustreError: Skipped 1 previous similar message
[ 2299.872304] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 192.168.204.148@tcp) was lost; in progress operations using this service will fail
[ 2310.058589] Lustre: Evicted from MGS (at 192.168.204.148@tcp) after server handle changed from 0x5ca00a437328bc5b to 0x5ca00a4373290691
[ 2327.986467] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2330.352383] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2417.440168] Lustre: 2367:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790921451/real 1790921451]  req@ffffa06f494e7100 x1877914786422400/t0(0) o400->MGC192.168.204.148@tcp@192.168.204.148@tcp:26/25 lens 224/224 e 0 to 1 dl 1790921467 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2417.499826] Lustre: 2367:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 16 previous similar messages
[ 2417.522980] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 192.168.204.148@tcp) was lost; in progress operations using this service will fail
[ 2417.550686] Lustre: Evicted from MGS (at 192.168.204.148@tcp) after server handle changed from 0x5ca00a4373290691 to 0x5ca00a43732a86c6
[ 2418.692040] Lustre: 15915:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.148@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2444.624908] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2447.425726] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2530.272910] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 192.168.204.148@tcp) was lost; in progress operations using this service will fail
[ 2540.466304] Lustre: Evicted from MGS (at 192.168.204.148@tcp) after server handle changed from 0x5ca00a43732a86c6 to 0x5ca00a43732c1ef6
[ 2556.305209] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2558.376520] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2587.767370] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 02:13:56 (1790921636)
[ 2592.337652] LustreError: lustre-MDT0000-mdc-ffffa06f4a65b800: operation mds_reint to node 192.168.204.148@tcp failed: rc = -19
[ 2592.347587] LustreError: Skipped 4 previous similar messages
[ 2611.168239] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 192.168.204.148@tcp) was lost; in progress operations using this service will fail
[ 2611.210921] Lustre: Evicted from MGS (at 192.168.204.148@tcp) after server handle changed from 0x5ca00a43732c1ef6 to 0x5ca00a43732ce5a4
[ 2619.630789] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2660.927352] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2696.604793] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2738.040347] Lustre: 15915:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.148@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2749.346369] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2773.490520] Lustre: lustre-MDT0000-mdc-ffffa06f4a65b800: Connection to lustre-MDT0000 (at 192.168.204.148@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2773.498481] Lustre: Skipped 8 previous similar messages
[ 2790.880204] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 192.168.204.148@tcp) was lost; in progress operations using this service will fail
[ 2790.895039] LustreError: Skipped 3 previous similar messages
[ 2801.079766] Lustre: Evicted from MGS (at 192.168.204.148@tcp) after server handle changed from 0x5ca00a43732d48bc to 0x5ca00a43732de769
[ 2801.088984] Lustre: Skipped 3 previous similar messages
[ 2801.096774] Lustre: MGC192.168.204.148@tcp: Connection restored to 192.168.204.148@tcp (at 192.168.204.148@tcp)
[ 2801.111472] Lustre: Skipped 17 previous similar messages
[ 2802.096717] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2862.661620] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2924.974115] Lustre: 15915:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.148@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2963.940401] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 02:20:12 (1790922012)
[ 3022.539376] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3025.130925] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3309.191566] LustreError: lustre-OST0000-osc-ffffa06f4a65b800: operation ldlm_enqueue to node 192.168.204.148@tcp failed: rc = -19
[ 3309.199753] LustreError: Skipped 10 previous similar messages
[ 3351.777051] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3354.686426] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3639.606441] Lustre: lustre-OST0000-osc-ffffa06f4a65b800: Connection to lustre-OST0000 (at 192.168.204.148@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3639.634530] Lustre: Skipped 4 previous similar messages
[ 3665.315659] Lustre: lustre-OST0000-osc-ffffa06f4a65b800: Connection restored to 192.168.204.148@tcp (at 192.168.204.148@tcp)
[ 3665.350995] Lustre: Skipped 7 previous similar messages
[ 3687.954416] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3689.735636] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3937.465163] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 02:36:25 (1790922985)
[ 3954.656651] Lustre: 50555:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790922989/real 1790922989]  req@ffffa06f4a373480 x1877914806817280/t0(0) o101->lustre-MDT0000-mdc-ffffa06f4a65b800@192.168.204.148@tcp:12/10 lens 576/1152 e 0 to 1 dl 1790923005 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3954.720664] Lustre: 50555:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
[ 3964.524563] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 02:36:52 (1790923012)
[ 3990.892538] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 02:37:20 (1790923040)
[ 4016.338839] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 02:37:45 (1790923065)
[ 4016.819362] Lustre: Mounted lustre-client - version 2.17.59_2_g8e10c0d
[ 4047.840317] Lustre: 2370:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790923082/real 1790923082]  req@ffffa06f49a19f80 x1877914806835968/t0(0) o400->lustre-MDT0000-mdc-ffffa06f4a65b800@192.168.204.148@tcp:12/10 lens 224/224 e 0 to 1 dl 1790923098 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4047.843606] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 192.168.204.148@tcp) was lost; in progress operations using this service will fail
[ 4047.913571] Lustre: 2370:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 4047.922479] LustreError: Skipped 2 previous similar messages
[ 4058.093807] Lustre: Evicted from MGS (at 192.168.204.148@tcp) after server handle changed from 0x5ca00a43732f3ef5 to 0x5ca00a4373426b4c
[ 4058.108232] Lustre: Skipped 2 previous similar messages
[ 4072.845613] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4074.470276] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4076.538250] Lustre: Unmounted lustre-client
[ 4082.867893] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 02:38:51 (1790923131)
[ 4205.345100] Lustre: 2369:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790923239/real 1790923239]  req@ffffa06f49a18e00 x1877914806869760/t0(0) o4->lustre-OST0000-osc-ffffa06f4a65b800@192.168.204.148@tcp:6/4 lens 488/448 e 0 to 1 dl 1790923255 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4205.385611] Lustre: 2369:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 55 previous similar messages
[ 4252.640276] Lustre: lustre-OST0000-osc-ffffa06f4a65b800: Connection to lustre-OST0000 (at 192.168.204.148@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4252.656021] Lustre: Skipped 14 previous similar messages
[ 4269.047221] Lustre: lustre-OST0000-osc-ffffa06f4a65b800: Connection restored to 192.168.204.148@tcp (at 192.168.204.148@tcp)
[ 4269.052806] Lustre: Skipped 16 previous similar messages
[ 4380.084376] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 02:43:48 (1790923428)
[ 4428.898854] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 02:44:37 (1790923477)
[ 4431.797907] Lustre: Unmounted lustre-client
[ 4453.735361] Lustre: Mounted lustre-client - version 2.17.59_2_g8e10c0d
[ 4460.049476] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 02:45:09 (1790923509)
[ 4460.358918] LustreError: 56483:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4461.409098] LustreError: 56483:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4461.548493] Lustre: *** cfs_fail_loc=305, val=0***
[ 4484.980783] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 02:45:34 (1790923534)
[ 4485.553608] Lustre: Mounted lustre-client - version 2.17.59_2_g8e10c0d
[ 4487.725599] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4498.070827] Lustre: Unmounted lustre-client
[ 4505.526425] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 02:45:54 (1790923554)
[ 4591.365352] LustreError: lustre-MDT0000-mdc-ffffa06f49485800: operation mds_reint to node 192.168.204.148@tcp failed: rc = -19
[ 4591.371111] LustreError: Skipped 1 previous similar message
[ 4610.528151] Lustre: 2368:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790923644/real 1790923644]  req@ffffa06f4552fb80 x1877914809532160/t0(0) o400->MGC192.168.204.148@tcp@192.168.204.148@tcp:26/25 lens 224/224 e 0 to 1 dl 1790923660 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4610.558857] Lustre: 2368:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 92 previous similar messages
[ 4610.567371] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 192.168.204.148@tcp) was lost; in progress operations using this service will fail
[ 4619.754573] Lustre: Evicted from MGS (at 192.168.204.148@tcp) after server handle changed from 0x5ca00a4373428f2c to 0x5ca00a4373454353
[ 4770.318685] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 02:50:19 (1790923819)
[ 4775.453980] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4787.701177] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 192.168.204.148@tcp) was lost; in progress operations using this service will fail
[ 4787.729625] Lustre: Evicted from MGS (at 192.168.204.148@tcp) after server handle changed from 0x5ca00a4373454353 to 0x5ca00a43734a0fb0
[ 4787.787894] LustreError: lustre-MDT0000-mdc-ffffa06f49485800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4803.989541] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 02:50:53 (1790923853)
[ 4804.391561] Lustre: Mounted lustre-client - version 2.17.59_2_g8e10c0d
[ 4811.524946] LustreError: lustre-OST0000-osc-ffffa06f49485800: operation ldlm_enqueue to node 192.168.204.148@tcp failed: rc = -107
[ 4811.553278] LustreError: lustre-OST0000-osc-ffffa06f49485800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4822.700680] Lustre: Unmounted lustre-client
[ 4830.155875] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 02:51:19 (1790923879)
[ 4837.000895] LustreError: lustre-MDT0000-mdc-ffffa06f49485800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4837.017215] LustreError: 60875:0:(file.c:6243:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4844.108398] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 02:51:33 (1790923893)
[ 4844.216772] LustreError: 61573:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4848.294532] LustreError: lustre-MDT0000-mdc-ffffa06f49485800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4848.308838] LustreError: 61596:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4848.316938] LustreError: 61573:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=905
[ 4848.325468] LustreError: 61573:0:(import.c:762:ptlrpc_connect_import_locked()) already connecting
[ 4848.398768] LustreError: 61602:0:(file.c:6243:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4849.491528] LustreError: 61608:0:(file.c:6243:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4849.506052] LustreError: 61608:0:(file.c:6243:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4851.777884] LustreError: 61630:0:(file.c:6243:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4851.789398] LustreError: 61630:0:(file.c:6243:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4856.386431] LustreError: 61675:0:(file.c:6243:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4856.393467] LustreError: 61675:0:(file.c:6243:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4865.071817] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 02:51:54 (1790923914)
[ 4874.223932] Lustre: lustre-OST0000-osc-ffffa06f49485800: Connection to lustre-OST0000 (at 192.168.204.148@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4874.241413] Lustre: Skipped 17 previous similar messages
[ 4901.562203] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4902.779716] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4912.716302] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 02:52:41 (1790923961)
[ 4936.701868] Lustre: lustre-OST0000-osc-ffffa06f49485800: Connection restored to 192.168.204.148@tcp (at 192.168.204.148@tcp)
[ 4936.726935] Lustre: Skipped 17 previous similar messages
[ 4950.792609] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4952.668093] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4963.446720] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 02:53:32 (1790924012)
[ 5030.810529] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5032.576590] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5040.771330] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 02:54:49 (1790924089)
[ 5079.442880] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5081.064870] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5087.189825] Lustre: Unmounted lustre-client
[ 5138.064935] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5140.579107] Lustre: Mounted lustre-client - version 2.17.59_2_g8e10c0d
[ 5146.262935] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 02:56:35 (1790924195)
[ 5148.224156] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5149.711923] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 02:56:38 (1790924198)
[ 5178.811948] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 02:57:07 (1790924227)
[ 5180.162975] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5182.021886] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 02:57:10 (1790924230)
[ 5183.422270] Lustre: *** cfs_fail_loc=805, val=0***
[ 5183.488132] Lustre: Mounted lustre-client - version 2.17.59_2_g8e10c0d
[ 5188.525074] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5209.064557] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 192.168.204.148@tcp) was lost; in progress operations using this service will fail
[ 5209.085691] Lustre: Evicted from MGS (at 192.168.204.148@tcp) after server handle changed from 0x5ca00a43734a1a37 to 0x5ca00a43734a1c52
[ 5210.736806] Lustre: 68240:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.148@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 5215.177209] Lustre: 2369:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790924249/real 1790924249]  req@ffffa06f49b0df80 x1877914811590272/t0(0) o400->lustre-MDT0000-mdc-ffffa06f4acd2000@192.168.204.148@tcp:12/10 lens 224/224 e 0 to 1 dl 1790924265 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5215.217388] Lustre: 2369:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 5226.607948] Lustre: Unmounted lustre-client
[ 5234.245447] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 02:58:03 (1790924283)
[ 5260.821463] LustreError: 2366:0:(client.c:3449:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffffa06f4b3adc00 x1877914811601792/t90194313220(90194313220) o101->lustre-MDT0000-mdc-ffffa06f4acd2000@192.168.204.148@tcp:12/10 lens 648/608 e 0 to 0 dl 1790924327 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5269.129160] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5270.492584] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5279.187302] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 02:58:48 (1790924328)
[ 5280.492163] LustreError: lustre-OST0000-osc-ffffa06f4acd2000: operation ost_write to node 192.168.204.148@tcp failed: rc = -107
[ 5280.502296] LustreError: Skipped 2 previous similar messages
[ 5280.514087] LustreError: lustre-OST0000-osc-ffffa06f4acd2000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5280.534379] Lustre: 2369:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.148@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5280.562485] LustreError: 72777:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0000-osc-ffffa06f4acd2000: namespace resource [0x240000400:0x3ca2:0x0].0x0 (ffffa06f4542b700) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5288.609584] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 02:58:57 (1790924337)
[ 5290.210949] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5292.056901] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 02:59:01 (1790924341)
[ 5293.767948] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5295.303845] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 02:59:04 (1790924344)
[ 5296.835069] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5298.359348] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 02:59:07 (1790924347)
[ 5299.932103] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5301.577209] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 02:59:10 (1790924350)
[ 5303.377550] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5305.227436] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 02:59:14 (1790924354)
[ 5306.950572] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5308.566362] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 02:59:17 (1790924357)
[ 5310.364572] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5312.180327] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 02:59:21 (1790924361)
[ 5313.773833] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5315.566789] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 02:59:24 (1790924364)
[ 5317.121993] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5318.547399] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 02:59:27 (1790924367)
[ 5320.077690] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5321.748505] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 02:59:30 (1790924370)
[ 5323.353258] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5325.206652] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 02:59:34 (1790924374)
[ 5327.650586] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5329.306665] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 02:59:38 (1790924378)
[ 5362.280792] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 03:00:11 (1790924411)
[ 5392.112187] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 03:00:41 (1790924441)
[ 5392.593930] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5402.925946] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 03:00:51 (1790924451)
[ 5403.355270] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5413.698516] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 03:01:02 (1790924462)
[ 5413.951679] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5422.197256] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 03:01:11 (1790924471)
[ 5422.447075] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5429.715937] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 03:01:18 (1790924478)
[ 5429.957150] Lustre: *** cfs_fail_loc=510, val=3***
[ 5437.111604] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 03:01:26 (1790924486)
[ 5437.420171] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5447.653285] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 03:01:36 (1790924496)
[ 5448.080651] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5507.554871] Lustre: lustre-OST0001-osc-ffffa06f4acd2000: Connection to lustre-OST0001 (at 192.168.204.148@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5507.582355] Lustre: Skipped 9 previous similar messages
[ 5513.961549] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 03:02:43 (1790924563)
[ 5514.315289] LustreError: 82991:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5517.146182] LustreError: lustre-MDT0000-mdc-ffffa06f4acd2000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5517.159586] LustreError: 83005:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffffa06f4acd2000: inode [0x20000a042:0xb:0x0] mdc close failed: rc = -108
[ 5517.198494] LustreError: 83005:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-MDT0000-mdc-ffffa06f4acd2000: namespace resource [0x20000a042:0x11:0x0].0xc (ffffa06f4542b900) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5517.238682] LustreError: 83010:0:(file.c:6243:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5517.247176] LustreError: 83010:0:(file.c:6243:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5518.400142] LustreError: 82991:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5520.523424] LustreError: 83021:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5523.304272] LustreError: lustre-MDT0000-mdc-ffffa06f4acd2000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5523.327121] LustreError: 83035:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-MDT0000-mdc-ffffa06f4acd2000: namespace resource [0x20000a042:0x11:0x0].0xc (ffffa06f4542a700) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5523.359176] LustreError: 83035:0:(ldlm_resource.c:1227:ldlm_resource_complain()) Skipped 2 previous similar messages
[ 5524.600305] LustreError: 83021:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5531.470134] LustreError: lustre-MDT0000-mdc-ffffa06f4acd2000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5531.829861] LustreError: 83076:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5534.696149] LustreError: lustre-MDT0000-mdc-ffffa06f4acd2000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5534.755593] LustreError: 83096:0:(file.c:6243:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5535.904260] LustreError: 83076:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5535.911482] LustreError: 83076:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffffa06f4acd2000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5536.980498] LustreError: 83113:0:(file.c:6243:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5537.000704] LustreError: 83113:0:(file.c:6243:ll_inode_revalidate_fini()) Skipped 12 previous similar messages
[ 5538.727304] Lustre: lustre-MDT0000-mdc-ffffa06f4acd2000: Connection restored to 192.168.204.148@tcp (at 192.168.204.148@tcp)
[ 5538.736843] Lustre: Skipped 13 previous similar messages
[ 5541.798636] LustreError: 83151:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5544.734485] LustreError: lustre-MDT0000-mdc-ffffa06f4acd2000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5544.759191] LustreError: 83165:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-MDT0000-mdc-ffffa06f4acd2000: namespace resource [0x20000a042:0x11:0x0].0xc (ffffa06f455f0a00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5544.800630] LustreError: 83165:0:(ldlm_resource.c:1227:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5545.872812] LustreError: 83151:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5552.854493] LustreError: lustre-MDT0000-mdc-ffffa06f4acd2000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5553.269595] LustreError: 83207:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5556.178831] LustreError: lustre-MDT0000-mdc-ffffa06f4acd2000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5556.191847] LustreError: 83221:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-MDT0000-mdc-ffffa06f4acd2000: namespace resource [0x20000a042:0x11:0x0].0xc (ffffa06f4542ae00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5556.207519] LustreError: 83221:0:(ldlm_resource.c:1227:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5557.344125] LustreError: 83207:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5560.337357] LustreError: lustre-MDT0000-mdc-ffffa06f4acd2000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5560.427394] LustreError: 83254:0:(file.c:6243:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5560.439399] LustreError: 83254:0:(file.c:6243:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5561.586355] LustreError: 83234:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffffa06f4acd2000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5569.501791] LustreError: 83306:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5569.506810] LustreError: 83306:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5573.183366] LustreError: lustre-MDT0000-mdc-ffffa06f4acd2000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5573.282516] LustreError: 83328:0:(file.c:6243:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5573.295578] LustreError: 83328:0:(file.c:6243:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5573.584166] LustreError: 83306:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5573.597456] LustreError: 83306:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5573.614611] LustreError: 83306:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffffa06f4acd2000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5586.772542] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 03:03:55 (1790924635)
[ 5607.735600] LustreError: 68238:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5607.740245] LustreError: 68238:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5611.738337] LustreError: 68238:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5611.756950] LustreError: 68238:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5619.717796] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 03:04:28 (1790924668)
[ 5670.844354] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 03:05:19 (1790924719)
[ 5735.264699] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 03:06:24 (1790924784)
[ 5769.679924] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 03:06:58 (1790924818)
[ 5770.306677] Lustre: Mounted lustre-client - version 2.17.59_2_g8e10c0d
[ 5893.794137] Lustre: Unmounted lustre-client
[ 5901.068658] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 03:09:09 (1790924949)
[ 5901.628488] LustreError: 87634:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5904.096326] LustreError: lustre-OST0000-osc-ffffa06f4acd2000: operation ldlm_enqueue to node 192.168.204.148@tcp failed: rc = -107
[ 5904.104558] LustreError: Skipped 9 previous similar messages
[ 5904.118257] LustreError: lustre-OST0000-osc-ffffa06f4acd2000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5904.126279] LustreError: 87716:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0000-osc-ffffa06f4acd2000: namespace resource [0x240000400:0x3ccd:0x0].0x0 (ffffa06f43bf9a00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5904.139911] LustreError: 87716:0:(ldlm_resource.c:1227:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5904.568142] LustreError: 87634:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5904.579701] Lustre: 2369:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.148@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5910.734843] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 03:09:19 (1790924959)
[ 5967.840288] Lustre: 88300:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790924963/real 1790924963]  req@ffffa06f4b1e5180 x1877914811773824/t0(0) o101->lustre-MDT0000-mdc-ffffa06f4acd2000@192.168.204.148@tcp:12/10 lens 328/344 e 0 to 1 dl 1790925018 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 5967.898221] Lustre: 88300:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 18 previous similar messages
[ 5975.294552] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 03:10:24 (1790925024)
[ 5977.212955] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 5978.968515] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 03:10:27 (1790925027)
[ 6042.001884] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6045.024332] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 03:11:33 (1790925093)
[ 6103.404413] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 03:12:32 (1790925152)
[ 6105.903526] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6107.860963] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 03:12:36 (1790925156)
[ 6109.873489] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6111.199426] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 03:12:40 (1790925160)
[ 6149.804694] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 03:13:18 (1790925198)
[ 6162.413752] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6190.048275] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 192.168.204.148@tcp) was lost; in progress operations using this service will fail
[ 6190.068328] LustreError: Skipped 2 previous similar messages
[ 6197.216459] Lustre: lustre-MDT0000-mdc-ffffa06f4acd2000: Connection to lustre-MDT0000 (at 192.168.204.148@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6197.232970] Lustre: Skipped 16 previous similar messages
[ 6200.367054] Lustre: Evicted from MGS (at 192.168.204.148@tcp) after server handle changed from 0x5ca00a43734a28ae to 0x5ca00a43734a40a9
[ 6200.382055] Lustre: Skipped 2 previous similar messages
[ 6200.397933] Lustre: MGC192.168.204.148@tcp: Connection restored to 192.168.204.148@tcp (at 192.168.204.148@tcp)
[ 6200.413977] Lustre: Skipped 11 previous similar messages
[ 6218.423884] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6219.942855] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6231.289618] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 03:14:40 (1790925280)
[ 6234.306539] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6236.653520] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 03:14:45 (1790925285)
[ 6253.560196] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 192.168.204.148@tcp) was lost; in progress operations using this service will fail
[ 6253.598422] Lustre: Evicted from MGS (at 192.168.204.148@tcp) after server handle changed from 0x5ca00a43734a40a9 to 0x5ca00a43734a455d
[ 6265.496488] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 03:15:14 (1790925314)
[ 6317.529749] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 03:16:06 (1790925366)
[ 6381.424723] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6385.612880] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6470.114546] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 192.168.204.148@tcp) was lost; in progress operations using this service will fail
[ 6470.126596] LustreError: Skipped 1 previous similar message
[ 6480.382297] Lustre: Evicted from MGS (at 192.168.204.148@tcp) after server handle changed from 0x5ca00a43734a476a to 0x5ca00a43734a5738
[ 6480.397010] Lustre: Skipped 1 previous similar message
[ 6494.536946] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6496.255751] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6533.679624] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6535.439849] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6571.360803] Lustre: 2370:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790925561/real 1790925561]  req@ffffa06f6d966680 x1877914818304000/t0(0) o400->lustre-MDT0000-mdc-ffffa06f4acd2000@192.168.204.148@tcp:12/10 lens 224/224 e 0 to 1 dl 1790925621 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6571.418463] Lustre: 2370:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 27 previous similar messages
[ 6671.896683] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 03:22:00 (1790925720)
[ 6681.559193] LustreError: lustre-OST0000-osc-ffffa06f4acd2000: operation ost_setattr to node 192.168.204.148@tcp failed: rc = -19
[ 6681.579307] LustreError: Skipped 24 previous similar messages
[ 6732.460248] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6735.557245] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6983.483994] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 03:27:12 (1790926032)
[ 7131.625566] Lustre: lustre-MDT0000-mdc-ffffa06f4acd2000: Connection to lustre-MDT0000 (at 192.168.204.148@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7131.652122] Lustre: Skipped 6 previous similar messages
[ 7141.871120] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 192.168.204.148@tcp) was lost; in progress operations using this service will fail
[ 7141.884588] LustreError: Skipped 1 previous similar message
[ 7141.902689] Lustre: Evicted from MGS (at 192.168.204.148@tcp) after server handle changed from 0x5ca00a43734a5d20 to 0x5ca00a437359e1a7
[ 7141.914984] Lustre: Skipped 1 previous similar message
[ 7141.921800] Lustre: MGC192.168.204.148@tcp: Connection restored to 192.168.204.148@tcp (at 192.168.204.148@tcp)
[ 7141.934738] Lustre: Skipped 9 previous similar messages
[ 7181.254728] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 03:30:30 (1790926230)
[ 7183.142625] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7185.073731] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 03:30:33 (1790926233)
[ 7187.104521] LustreError: lustre-MDT0000-mdc-ffffa06f4acd2000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7195.169771] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 03:30:44 (1790926244)
[ 7362.549677] LustreError: lustre-OST0000-osc-ffffa06f4acd2000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7365.144662] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 03:33:33 (1790926413)
[ 7391.200699] Lustre: 2369:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790926421/real 1790926421]  req@ffffa06f713a1880 x1877914829380608/t0(0) o4->lustre-OST0000-osc-ffffa06f4acd2000@192.168.204.148@tcp:6/4 lens 4584/448 e 0 to 1 dl 1790926441 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7391.268136] Lustre: 2369:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 7410.628117] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 03:34:19 (1790926459)
[ 7412.364392] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7414.465291] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 03:34:23 (1790926463)
[ 7416.070687] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7418.018781] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 03:34:26 (1790926466)
[ 7452.298402] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 03:35:01 (1790926501)
[ 7493.606433] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 192.168.204.148@tcp) was lost; in progress operations using this service will fail
[ 7493.643675] Lustre: Evicted from MGS (at 192.168.204.148@tcp) after server handle changed from 0x5ca00a437359e1a7 to 0x5ca00a43735a1563
[ 7493.653674] Lustre: Skipped 1 previous similar message
[ 7508.862963] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 03:35:57 (1790926557)
[ 7510.522187] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7512.392520] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 03:36:01 (1790926561)
[ 7514.132975] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7515.982472] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 03:36:04 (1790926564)
[ 7528.074230] Lustre: Unmounted lustre-client
[ 7530.330888] Lustre: Mounted lustre-client - version 2.17.59_2_g8e10c0d
[ 7533.499208] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7568.253787] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2