[ 1935.132293] LustreError: lustre-OST0000-osc-ffff932a917b8800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1935.152148] Lustre: 2353:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.147@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1944.300823] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 08:19:03 (1787833143)
[ 1948.191650] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1951.021813] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 08:19:10 (1787833150)
[ 1953.398875] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1955.449804] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 08:19:15 (1787833155)
[ 1960.706799] LustreError: lustre-MDT0000-mdc-ffff932a917b8800: operation mds_reint to node 192.168.201.147@tcp failed: rc = -19
[ 1960.734232] LustreError: Skipped 1 previous similar message
[ 1978.848937] LustreError: MGC192.168.201.147@tcp: Connection to MGS (at 192.168.201.147@tcp) was lost; in progress operations using this service will fail
[ 1989.037460] Lustre: Evicted from MGS (at 192.168.201.147@tcp) after server handle changed from 0xd3a05e4b20f55b5e to 0xd3a05e4b20f57233
[ 1990.719547] Lustre: 15933:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.147@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2094.411419] LustreError: lustre-MDT0000-mdc-ffff932a917b8800: operation ldlm_enqueue to node 192.168.201.147@tcp failed: rc = -19
[ 2110.945415] LustreError: MGC192.168.201.147@tcp: Connection to MGS (at 192.168.201.147@tcp) was lost; in progress operations using this service will fail
[ 2121.192758] Lustre: Evicted from MGS (at 192.168.201.147@tcp) after server handle changed from 0xd3a05e4b20f57233 to 0xd3a05e4b20f76d58
[ 2139.139970] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 08:22:18 (1787833338)
[ 2140.072281] Lustre: *** cfs_fail_loc=305, val=0***
[ 2140.075616] Lustre: Skipped 2 previous similar messages
[ 2182.626346] LustreError: MGC192.168.201.147@tcp: Connection to MGS (at 192.168.201.147@tcp) was lost; in progress operations using this service will fail
[ 2191.847750] Lustre: lustre-MDT0000-mdc-ffff932a917b8800: Connection to lustre-MDT0000 (at 192.168.201.147@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2191.851242] Lustre: Evicted from MGS (at 192.168.201.147@tcp) after server handle changed from 0xd3a05e4b20f76d58 to 0xd3a05e4b20f7711e
[ 2191.863767] Lustre: Skipped 12 previous similar messages
[ 2191.884675] Lustre: MGC192.168.201.147@tcp: Connection restored to 192.168.201.147@tcp (at 192.168.201.147@tcp)
[ 2191.892124] Lustre: Skipped 14 previous similar messages
[ 2206.144772] Lustre: DEBUG MARKER: oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2207.720361] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2217.234740] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 08:23:36 (1787833416)
[ 2232.822804] LustreError: MGC192.168.201.147@tcp: Connection to MGS (at 192.168.201.147@tcp) was lost; in progress operations using this service will fail
[ 2232.862532] Lustre: Evicted from MGS (at 192.168.201.147@tcp) after server handle changed from 0xd3a05e4b20f7711e to 0xd3a05e4b20f77618
[ 2232.928775] LustreError: lustre-MDT0000-mdc-ffff932a917b8800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2257.513337] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 08:24:16 (1787833456)
[ 2274.191657] LustreError: lustre-OST0000-osc-ffff932a917b8800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2297.454271] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 08:24:57 (1787833497)
[ 2310.739549] LustreError: lustre-MDT0000-mdc-ffff932a917b8800: operation mds_reint to node 192.168.201.147@tcp failed: rc = -19
[ 2330.081455] LustreError: MGC192.168.201.147@tcp: Connection to MGS (at 192.168.201.147@tcp) was lost; in progress operations using this service will fail
[ 2330.115832] Lustre: Evicted from MGS (at 192.168.201.147@tcp) after server handle changed from 0xd3a05e4b20f77618 to 0xd3a05e4b20f7c60c
[ 2358.654771] Lustre: DEBUG MARKER: oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2361.662279] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2445.807777] LustreError: MGC192.168.201.147@tcp: Connection to MGS (at 192.168.201.147@tcp) was lost; in progress operations using this service will fail
[ 2445.864393] Lustre: Evicted from MGS (at 192.168.201.147@tcp) after server handle changed from 0xd3a05e4b20f7c60c to 0xd3a05e4b20f982c4
[ 2462.011823] Lustre: DEBUG MARKER: oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2464.244427] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2549.793717] Lustre: 2353:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787833734/real 1787833734]  req@ffff932a99a6a1c0 x1874676943110528/t0(0) o400->MGC192.168.201.147@tcp@192.168.201.147@tcp:26/25 lens 224/224 e 0 to 1 dl 1787833750 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2549.810588] Lustre: 2353:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 15 previous similar messages
[ 2549.814531] LustreError: MGC192.168.201.147@tcp: Connection to MGS (at 192.168.201.147@tcp) was lost; in progress operations using this service will fail
[ 2560.055383] Lustre: Evicted from MGS (at 192.168.201.147@tcp) after server handle changed from 0xd3a05e4b20f982c4 to 0xd3a05e4b20fb53f7
[ 2576.507558] Lustre: DEBUG MARKER: oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2579.129886] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2609.286354] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 08:30:08 (1787833808)
[ 2613.768204] LustreError: lustre-MDT0000-mdc-ffff932a917b8800: operation ldlm_enqueue to node 192.168.201.147@tcp failed: rc = -19
[ 2613.781651] LustreError: Skipped 5 previous similar messages
[ 2634.731586] LustreError: MGC192.168.201.147@tcp: Connection to MGS (at 192.168.201.147@tcp) was lost; in progress operations using this service will fail
[ 2634.770995] Lustre: Evicted from MGS (at 192.168.201.147@tcp) after server handle changed from 0xd3a05e4b20fb53f7 to 0xd3a05e4b20fc3387
[ 2635.832072] Lustre: 15933:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.147@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2648.066627] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2679.280791] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2715.478798] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2757.020908] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2796.001435] LustreError: MGC192.168.201.147@tcp: Connection to MGS (at 192.168.201.147@tcp) was lost; in progress operations using this service will fail
[ 2796.016630] LustreError: Skipped 3 previous similar messages
[ 2806.194828] Lustre: Evicted from MGS (at 192.168.201.147@tcp) after server handle changed from 0xd3a05e4b20fc95cd to 0xd3a05e4b20fceb8d
[ 2806.202548] Lustre: Skipped 3 previous similar messages
[ 2806.218315] Lustre: MGC192.168.201.147@tcp: Connection restored to 192.168.201.147@tcp (at 192.168.201.147@tcp)
[ 2806.231447] Lustre: Skipped 17 previous similar messages
[ 2807.902151] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2836.020646] Lustre: lustre-MDT0000-mdc-ffff932a917b8800: Connection to lustre-MDT0000 (at 192.168.201.147@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2836.042876] Lustre: Skipped 9 previous similar messages
[ 2866.790301] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2920.504641] Lustre: 15933:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.147@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2961.970385] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 08:36:01 (1787834161)
[ 3018.214420] Lustre: DEBUG MARKER: oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3020.690160] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3306.978035] LustreError: lustre-OST0000-osc-ffff932a917b8800: operation ldlm_enqueue to node 192.168.201.147@tcp failed: rc = -107
[ 3306.988422] LustreError: Skipped 8 previous similar messages
[ 3348.351494] Lustre: DEBUG MARKER: oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3350.674757] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3637.985440] Lustre: lustre-OST0000-osc-ffff932a917b8800: Connection to lustre-OST0000 (at 192.168.201.147@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3638.012286] Lustre: Skipped 3 previous similar messages
[ 3680.594316] Lustre: DEBUG MARKER: oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3683.056484] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3931.660802] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 08:52:11 (1787835131)
[ 3949.536174] Lustre: 50590:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787835134/real 1787835134]  req@ffff932a89855e00 x1874676964407680/t0(0) o101->lustre-MDT0000-mdc-ffff932a917b8800@192.168.201.147@tcp:12/10 lens 576/1152 e 0 to 1 dl 1787835150 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3949.581753] Lustre: 50590:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
[ 3949.681804] Lustre: lustre-MDT0000-mdc-ffff932a917b8800: Connection restored to 192.168.201.147@tcp (at 192.168.201.147@tcp)
[ 3949.704234] Lustre: Skipped 6 previous similar messages
[ 3957.380826] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 08:52:37 (1787835157)
[ 3982.922389] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 08:53:03 (1787835183)
[ 4007.696587] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 08:53:27 (1787835207)
[ 4008.038933] Lustre: Mounted lustre-client - version 2.17.57_60_g13a1c7b
[ 4040.096248] Lustre: 2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787835225/real 1787835225]  req@ffff932a90f4e940 x1874676964426112/t0(0) o400->MGC192.168.201.147@tcp@192.168.201.147@tcp:26/25 lens 224/224 e 0 to 1 dl 1787835241 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4040.146437] Lustre: 2355:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 4040.159669] LustreError: MGC192.168.201.147@tcp: Connection to MGS (at 192.168.201.147@tcp) was lost; in progress operations using this service will fail
[ 4040.173259] LustreError: Skipped 2 previous similar messages
[ 4040.221570] Lustre: Evicted from MGS (at 192.168.201.147@tcp) after server handle changed from 0xd3a05e4b20fe6a3a to 0xd3a05e4b211457bc
[ 4040.233632] Lustre: Skipped 2 previous similar messages
[ 4055.823959] Lustre: DEBUG MARKER: oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4057.445616] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4059.701967] Lustre: Unmounted lustre-client
[ 4066.129914] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 08:54:25 (1787835265)
[ 4205.024129] Lustre: 2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787835390/real 1787835390]  req@ffff932a90f4c780 x1874676964460288/t0(0) o4->lustre-OST0000-osc-ffff932a917b8800@192.168.201.147@tcp:6/4 lens 488/448 e 0 to 1 dl 1787835406 ref 2 fl Rpc:XQr/602/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4205.045251] Lustre: 2355:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 76 previous similar messages
[ 4253.152536] Lustre: lustre-OST0000-osc-ffff932a917b8800: Connection to lustre-OST0000 (at 192.168.201.147@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4253.178169] Lustre: Skipped 15 previous similar messages
[ 4362.088256] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 08:59:21 (1787835561)
[ 4411.469394] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 09:00:11 (1787835611)
[ 4414.473915] Lustre: Unmounted lustre-client
[ 4438.974698] Lustre: Mounted lustre-client - version 2.17.57_60_g13a1c7b
[ 4446.483330] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 09:00:46 (1787835646)
[ 4446.973375] LustreError: 56518:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4448.037465] LustreError: 56518:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4448.389342] Lustre: *** cfs_fail_loc=305, val=0***
[ 4471.168358] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 09:01:11 (1787835671)
[ 4471.665762] Lustre: Mounted lustre-client - version 2.17.57_60_g13a1c7b
[ 4474.253554] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4484.539214] Lustre: Unmounted lustre-client
[ 4490.517563] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 09:01:30 (1787835690)
[ 4577.555363] LustreError: lustre-MDT0000-mdc-ffff932a83673800: operation mds_reint to node 192.168.201.147@tcp failed: rc = -19
[ 4577.567943] LustreError: Skipped 1 previous similar message
[ 4596.704234] Lustre: 2352:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787835781/real 1787835781]  req@ffff932a86d9d680 x1874676967136256/t0(0) o400->MGC192.168.201.147@tcp@192.168.201.147@tcp:26/25 lens 224/224 e 0 to 1 dl 1787835797 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4596.743644] Lustre: 2352:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 76 previous similar messages
[ 4596.759175] LustreError: MGC192.168.201.147@tcp: Connection to MGS (at 192.168.201.147@tcp) was lost; in progress operations using this service will fail
[ 4596.784629] Lustre: Evicted from MGS (at 192.168.201.147@tcp) after server handle changed from 0xd3a05e4b211471f5 to 0xd3a05e4b211725dd
[ 4596.796496] Lustre: MGC192.168.201.147@tcp: Connection restored to 192.168.201.147@tcp (at 192.168.201.147@tcp)
[ 4596.805551] Lustre: Skipped 23 previous similar messages
[ 4777.022602] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 09:06:16 (1787835976)
[ 4783.866382] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4804.609934] LustreError: MGC192.168.201.147@tcp: Connection to MGS (at 192.168.201.147@tcp) was lost; in progress operations using this service will fail
[ 4804.664823] Lustre: Evicted from MGS (at 192.168.201.147@tcp) after server handle changed from 0xd3a05e4b211725dd to 0xd3a05e4b211bf11b
[ 4804.785321] LustreError: lustre-MDT0000-mdc-ffff932a83673800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4823.033546] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 09:07:02 (1787836022)
[ 4823.470858] Lustre: Mounted lustre-client - version 2.17.57_60_g13a1c7b
[ 4831.170054] LustreError: lustre-OST0000-osc-ffff932a83673800: operation ldlm_enqueue to node 192.168.201.147@tcp failed: rc = -107
[ 4831.183046] LustreError: lustre-OST0000-osc-ffff932a83673800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4843.604911] Lustre: Unmounted lustre-client
[ 4853.025196] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 09:07:32 (1787836052)
[ 4857.584797] Lustre: lustre-MDT0000-mdc-ffff932a83673800: Connection to lustre-MDT0000 (at 192.168.201.147@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4857.605932] Lustre: Skipped 13 previous similar messages
[ 4860.848110] LustreError: lustre-MDT0000-mdc-ffff932a83673800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4860.874203] LustreError: 61009:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff932a83673800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff932a99be2400) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4860.927135] LustreError: 60930:0:(file.c:6176:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4869.761673] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 09:07:48 (1787836068)
[ 4870.053794] LustreError: 61631:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4874.506682] LustreError: lustre-MDT0000-mdc-ffff932a83673800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4874.533536] LustreError: 61648:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4874.559417] LustreError: 61631:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=503
[ 4874.575425] LustreError: 61631:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting
[ 4874.646598] LustreError: 61653:0:(file.c:6176:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4875.858094] LustreError: 61659:0:(file.c:6176:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4875.875723] LustreError: 61659:0:(file.c:6176:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4878.293983] LustreError: 61681:0:(file.c:6176:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4878.305956] LustreError: 61681:0:(file.c:6176:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4883.118480] LustreError: 61726:0:(file.c:6176:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4883.144279] LustreError: 61726:0:(file.c:6176:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4893.028909] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 09:08:13 (1787836093)
[ 4935.633537] Lustre: DEBUG MARKER: oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4937.730794] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4949.351645] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 09:09:08 (1787836148)
[ 4995.209571] Lustre: DEBUG MARKER: oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4997.646302] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5008.310595] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 09:10:08 (1787836208)
[ 5074.748559] Lustre: DEBUG MARKER: oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5075.983991] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5085.021714] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 09:11:24 (1787836284)
[ 5128.970549] Lustre: DEBUG MARKER: oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5130.887424] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5136.429430] Lustre: Unmounted lustre-client
[ 5193.657846] Lustre: DEBUG MARKER: oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5196.318711] Lustre: Mounted lustre-client - version 2.17.57_60_g13a1c7b
[ 5203.180691] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 09:13:22 (1787836402)
[ 5206.093932] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5208.189334] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 09:13:27 (1787836407)
[ 5242.678619] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 09:14:02 (1787836442)
[ 5244.364487] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5246.350941] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 09:14:05 (1787836445)
[ 5248.127862] Lustre: *** cfs_fail_loc=805, val=0***
[ 5248.185236] Lustre: Mounted lustre-client - version 2.17.57_60_g13a1c7b
[ 5254.186496] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5274.016223] Lustre: 2352:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787836459/real 1787836459]  req@ffff932a86c9b480 x1874676969193344/t0(0) o400->lustre-MDT0000-mdc-ffff932a91ddd000@192.168.201.147@tcp:12/10 lens 224/224 e 0 to 1 dl 1787836475 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5274.047744] Lustre: 2352:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5274.052288] LustreError: MGC192.168.201.147@tcp: Connection to MGS (at 192.168.201.147@tcp) was lost; in progress operations using this service will fail
[ 5284.340107] Lustre: Evicted from MGS (at 192.168.201.147@tcp) after server handle changed from 0xd3a05e4b211bfb0f to 0xd3a05e4b211bfd2a
[ 5284.362890] Lustre: MGC192.168.201.147@tcp: Connection restored to 192.168.201.147@tcp (at 192.168.201.147@tcp)
[ 5284.377140] Lustre: Skipped 12 previous similar messages
[ 5299.704618] LustreError: 2351:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff932a8817e000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff932a86ddb900) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5302.896310] Lustre: Unmounted lustre-client
[ 5313.417781] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 09:15:12 (1787836512)
[ 5347.956630] Lustre: 68295:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.147@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 5356.081439] LustreError: 2351:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff932a84aad680 x1874676969205120/t90194313221(90194313221) o101->lustre-MDT0000-mdc-ffff932a91ddd000@192.168.201.147@tcp:12/10 lens 648/608 e 0 to 0 dl 1787836573 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5366.615475] Lustre: DEBUG MARKER: oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5368.831873] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5378.184505] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 09:16:18 (1787836578)
[ 5379.637839] LustreError: lustre-OST0000-osc-ffff932a91ddd000: operation ost_write to node 192.168.201.147@tcp failed: rc = -107
[ 5379.647684] LustreError: Skipped 2 previous similar messages
[ 5379.666415] LustreError: lustre-OST0000-osc-ffff932a91ddd000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5379.691366] Lustre: 2354:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.147@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5379.718444] LustreError: 72836:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff932a91ddd000: namespace resource [0x240000400:0x4082:0x0].0x0 (ffff932a99be2800) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5379.734046] LustreError: 72836:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5388.333724] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 09:16:28 (1787836588)
[ 5390.230717] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5391.911433] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 09:16:31 (1787836591)
[ 5393.238423] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5395.122391] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 09:16:34 (1787836594)
[ 5396.560678] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5398.522833] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 09:16:38 (1787836598)
[ 5400.075733] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5401.826972] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 09:16:41 (1787836601)
[ 5403.158909] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5405.311345] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 09:16:44 (1787836604)
[ 5406.770558] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5408.456870] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 09:16:48 (1787836608)
[ 5410.281867] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5412.083288] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 09:16:51 (1787836611)
[ 5413.980736] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5415.925811] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 09:16:55 (1787836615)
[ 5417.843633] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5419.601553] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 09:16:59 (1787836619)
[ 5421.645254] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5423.196843] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 09:17:03 (1787836623)
[ 5425.025617] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5426.820405] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 09:17:06 (1787836626)
[ 5429.341473] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5431.391076] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 09:17:10 (1787836630)
[ 5469.688763] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 09:17:49 (1787836669)
[ 5500.012272] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 09:18:19 (1787836699)
[ 5500.585410] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5511.481391] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 09:18:31 (1787836711)
[ 5511.779418] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5523.278644] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 09:18:42 (1787836722)
[ 5523.601712] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5532.105805] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 09:18:52 (1787836732)
[ 5532.445985] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5542.120340] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 09:19:01 (1787836741)
[ 5542.422763] Lustre: *** cfs_fail_loc=510, val=3***
[ 5551.094503] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 09:19:10 (1787836750)
[ 5551.411134] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5610.914041] Lustre: lustre-OST0001-osc-ffff932a91ddd000: Connection to lustre-OST0001 (at 192.168.201.147@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5610.935903] Lustre: Skipped 12 previous similar messages
[ 5617.331230] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 09:20:17 (1787836817)
[ 5617.623361] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5684.060609] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 09:21:23 (1787836883)
[ 5684.302736] LustreError: 83056:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5687.118694] LustreError: lustre-MDT0000-mdc-ffff932a91ddd000: operation ldlm_enqueue to node 192.168.201.147@tcp failed: rc = -107
[ 5687.139103] LustreError: lustre-MDT0000-mdc-ffff932a91ddd000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5687.151462] LustreError: 83071:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff932a91ddd000: inode [0x20000a042:0xb:0x0] mdc close failed: rc = -108
[ 5687.175763] LustreError: 83071:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff932a91ddd000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff932a86ddbb00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5687.199577] LustreError: 83076:0:(file.c:6176:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5687.217315] LustreError: 83076:0:(file.c:6176:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5688.368143] LustreError: 83056:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5690.503986] LustreError: 83086:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5693.448272] LustreError: lustre-MDT0000-mdc-ffff932a91ddd000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5693.470246] LustreError: 83100:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff932a91ddd000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff932a99be2000) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5693.487556] LustreError: 83100:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5694.584094] LustreError: 83086:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5701.532488] LustreError: lustre-MDT0000-mdc-ffff932a91ddd000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5701.990868] LustreError: 83142:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5704.908361] LustreError: lustre-MDT0000-mdc-ffff932a91ddd000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5704.973716] LustreError: 83161:0:(file.c:6176:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5706.073056] LustreError: 83142:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5706.089741] LustreError: 83142:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff932a91ddd000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5706.106541] LustreError: 83142:0:(file.c:251:ll_close_inode_openhandle()) Skipped 1 previous similar message
[ 5707.207989] LustreError: 83179:0:(file.c:6176:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5707.228730] LustreError: 83179:0:(file.c:6176:ll_inode_revalidate_fini()) Skipped 12 previous similar messages
[ 5712.204682] LustreError: 83217:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5715.112694] LustreError: lustre-MDT0000-mdc-ffff932a91ddd000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5715.138560] LustreError: 83231:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff932a91ddd000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff932a86ee3800) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5715.171213] LustreError: 83231:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 2 previous similar messages
[ 5716.296132] LustreError: 83217:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5723.300713] LustreError: lustre-MDT0000-mdc-ffff932a91ddd000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5723.643802] LustreError: 83273:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5726.607944] LustreError: lustre-MDT0000-mdc-ffff932a91ddd000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5727.712486] LustreError: 83273:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5730.859300] LustreError: lustre-MDT0000-mdc-ffff932a91ddd000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5730.929779] LustreError: 83321:0:(file.c:6176:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5730.934765] LustreError: 83321:0:(file.c:6176:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5731.928700] LustreError: 83301:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff932a91ddd000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5739.977892] LustreError: 83373:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5739.994427] LustreError: 83373:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5743.637978] LustreError: lustre-MDT0000-mdc-ffff932a91ddd000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5743.732640] LustreError: 83395:0:(file.c:6176:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5743.743499] LustreError: 83395:0:(file.c:6176:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5744.073544] LustreError: 83373:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5744.083828] LustreError: 83373:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5744.102534] LustreError: 83373:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff932a91ddd000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5758.220132] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 09:22:37 (1787836957)
[ 5778.549490] LustreError: 68292:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5778.565286] LustreError: 68292:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5782.641047] LustreError: 68292:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5782.645053] LustreError: 68292:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5790.127868] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 09:23:10 (1787836990)
[ 5840.444821] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 09:24:00 (1787837040)
[ 5899.232246] Lustre: 85552:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787837044/real 1787837044]  req@ffff932a84aae580 x1874676969335424/t0(0) o101->lustre-MDT0000-mdc-ffff932a91ddd000@192.168.201.147@tcp:12/10 lens 576/1152 e 0 to 1 dl 1787837100 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'stat.0' uid:0 gid:0 projid:0
[ 5899.278760] Lustre: 85552:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 19 previous similar messages
[ 5899.295708] Lustre: lustre-MDT0000-mdc-ffff932a91ddd000: Connection restored to 192.168.201.147@tcp (at 192.168.201.147@tcp)
[ 5899.309525] Lustre: Skipped 17 previous similar messages
[ 5905.042361] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 09:25:04 (1787837104)
[ 5938.782987] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 09:25:38 (1787837138)
[ 5939.233947] Lustre: Mounted lustre-client - version 2.17.57_60_g13a1c7b
[ 6063.330894] Lustre: Unmounted lustre-client
[ 6069.600937] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 09:27:49 (1787837269)
[ 6070.300806] LustreError: 87699:0:(osc_request.c:3165:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 6072.904332] LustreError: lustre-OST0000-osc-ffff932a91ddd000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6072.924535] LustreError: 87786:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff932a91ddd000: namespace resource [0x240000400:0x40ad:0x0].0x0 (ffff932a99be2d00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6072.950872] LustreError: 87786:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 3 previous similar messages
[ 6073.161750] LustreError: 87699:0:(osc_request.c:3165:osc_build_rpc()) cfs_fail_timeout interrupted
[ 6073.172001] Lustre: 2352:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.147@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 6079.280791] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 09:27:59 (1787837279)
[ 6145.105334] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 09:29:04 (1787837344)
[ 6146.421597] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 6147.947917] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 09:29:07 (1787837347)
[ 6213.593602] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6214.980382] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 09:30:15 (1787837415)
[ 6272.993122] Lustre: lustre-MDT0000-mdc-ffff932a91ddd000: Connection to lustre-MDT0000 (at 192.168.201.147@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6273.010222] Lustre: Skipped 16 previous similar messages
[ 6274.575854] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 09:31:14 (1787837474)
[ 6276.459943] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6277.757132] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 09:31:17 (1787837477)
[ 6279.288298] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6280.574288] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 09:31:20 (1787837480)
[ 6316.116621] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 09:31:55 (1787837515)
[ 6330.110801] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6357.984267] LustreError: MGC192.168.201.147@tcp: Connection to MGS (at 192.168.201.147@tcp) was lost; in progress operations using this service will fail
[ 6358.011764] LustreError: Skipped 2 previous similar messages
[ 6358.053717] Lustre: Evicted from MGS (at 192.168.201.147@tcp) after server handle changed from 0xd3a05e4b211c0939 to 0xd3a05e4b211c22ed
[ 6358.062464] Lustre: Skipped 2 previous similar messages
[ 6373.891888] Lustre: DEBUG MARKER: oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6376.431649] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6388.446687] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 09:33:08 (1787837588)
[ 6391.269576] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6392.735992] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 09:33:12 (1787837592)
[ 6408.168490] LustreError: MGC192.168.201.147@tcp: Connection to MGS (at 192.168.201.147@tcp) was lost; in progress operations using this service will fail
[ 6408.206459] Lustre: Evicted from MGS (at 192.168.201.147@tcp) after server handle changed from 0xd3a05e4b211c22ed to 0xd3a05e4b211c2627
[ 6416.995813] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 09:33:36 (1787837616)
[ 6465.818651] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 09:34:25 (1787837665)
[ 6472.312579] LustreError: lustre-OST0000-osc-ffff932a91ddd000: operation ost_setattr to node 192.168.201.147@tcp failed: rc = -19
[ 6472.320038] LustreError: Skipped 9 previous similar messages
[ 6504.767103] Lustre: 2354:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787837645/real 1787837645]  req@ffff932a90f54780 x1874676969429888/t0(0) o400->lustre-MDT0000-mdc-ffff932a91ddd000@192.168.201.147@tcp:12/10 lens 224/224 e 0 to 1 dl 1787837700 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6504.803245] Lustre: 2354:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 14 previous similar messages
[ 6528.229694] Lustre: DEBUG MARKER: oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6531.187593] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6619.617655] LustreError: MGC192.168.201.147@tcp: Connection to MGS (at 192.168.201.147@tcp) was lost; in progress operations using this service will fail
[ 6619.641257] LustreError: Skipped 1 previous similar message
[ 6628.856528] Lustre: Evicted from MGS (at 192.168.201.147@tcp) after server handle changed from 0xd3a05e4b211c29ae to 0xd3a05e4b211c3799
[ 6628.877991] Lustre: Skipped 1 previous similar message
[ 6628.884500] Lustre: MGC192.168.201.147@tcp: Connection restored to 192.168.201.147@tcp (at 192.168.201.147@tcp)
[ 6628.890708] Lustre: Skipped 10 previous similar messages
[ 6643.026865] Lustre: DEBUG MARKER: oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6645.018803] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6698.326795] Lustre: DEBUG MARKER: oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6699.885531] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6839.618718] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 09:40:39 (1787838039)
[ 6901.761861] Lustre: DEBUG MARKER: oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6905.949312] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 7156.586689] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 09:45:56 (1787838356)
[ 7320.053056] Lustre: lustre-MDT0000-mdc-ffff932a91ddd000: Connection to lustre-MDT0000 (at 192.168.201.147@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7320.083380] Lustre: Skipped 7 previous similar messages
[ 7335.404206] LustreError: MGC192.168.201.147@tcp: Connection to MGS (at 192.168.201.147@tcp) was lost; in progress operations using this service will fail
[ 7335.412497] LustreError: Skipped 1 previous similar message
[ 7335.420770] Lustre: Evicted from MGS (at 192.168.201.147@tcp) after server handle changed from 0xd3a05e4b211c3a8d to 0xd3a05e4b212b703f
[ 7335.436837] Lustre: Skipped 1 previous similar message
[ 7335.444489] Lustre: MGC192.168.201.147@tcp: Connection restored to 192.168.201.147@tcp (at 192.168.201.147@tcp)
[ 7335.463109] Lustre: Skipped 3 previous similar messages
[ 7370.300615] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 09:49:30 (1787838570)
[ 7371.930087] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7373.805484] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 09:49:33 (1787838573)
[ 7376.405180] LustreError: lustre-MDT0000-mdc-ffff932a91ddd000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7376.453082] LustreError: 102049:0:(file.c:6176:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 7376.468768] LustreError: 102049:0:(file.c:6176:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 7384.979602] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 09:49:45 (1787838585)
[ 7387.294831] LustreError: lustre-OST0000-osc-ffff932a91ddd000: operation ost_statfs to node 192.168.201.147@tcp failed: rc = -107
[ 7387.311028] LustreError: Skipped 326 previous similar messages
[ 7552.053180] LustreError: lustre-OST0000-osc-ffff932a91ddd000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7554.388198] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 09:52:33 (1787838753)
[ 7580.576263] Lustre: 2354:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787838761/real 1787838761]  req@ffff932a849a0b40 x1874676986607104/t0(0) o4->lustre-OST0000-osc-ffff932a91ddd000@192.168.201.147@tcp:6/4 lens 4584/448 e 0 to 1 dl 1787838781 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7580.611139] Lustre: 2354:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 14 previous similar messages
[ 7597.929914] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 09:53:17 (1787838797)
[ 7599.154645] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7600.662638] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 09:53:20 (1787838800)
[ 7602.042196] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7603.671186] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 09:53:23 (1787838803)
[ 7638.179348] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 09:53:58 (1787838838)
[ 7674.854446] LustreError: MGC192.168.201.147@tcp: Connection to MGS (at 192.168.201.147@tcp) was lost; in progress operations using this service will fail
[ 7674.869447] Lustre: Evicted from MGS (at 192.168.201.147@tcp) after server handle changed from 0xd3a05e4b212b703f to 0xd3a05e4b212b9a54
[ 7674.877451] Lustre: Skipped 1 previous similar message
[ 7690.392798] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 09:54:50 (1787838890)
[ 7691.861817] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7693.463487] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 09:54:53 (1787838893)
[ 7695.011698] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7696.783913] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 09:54:56 (1787838896)
[ 7707.191721] Lustre: Unmounted lustre-client
[ 7709.199366] Lustre: Mounted lustre-client - version 2.17.57_60_g13a1c7b
[ 7711.699993] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7741.496880] Lustre: 106823:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.147@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 7753.549593] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2