[ 1966.172914] LustreError: lustre-OST0000-osc-ffff965ac668a800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1967.788528] Lustre: DEBUG MARKER: recovery-small test_24b: @@@@@@ IGNORE (bz5494): multiop didn't fail fsync: 5 or close: 0
[ 1970.619855] Lustre: DEBUG MARKER: recovery-small test_24b: @@@@@@ FAIL: no discarded dirty page found!
[ 1986.729331] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 16:29:21 (1787603361)
[ 1988.604345] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1990.410550] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 16:29:25 (1787603365)
[ 1992.144043] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1993.925357] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 16:29:28 (1787603368)
[ 1997.973174] LustreError: lustre-MDT0000-mdc-ffff965ac668a800: operation ldlm_enqueue to node 192.168.204.132@tcp failed: rc = -19
[ 1997.990543] LustreError: Skipped 1 previous similar message
[ 2015.712548] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 192.168.204.132@tcp) was lost; in progress operations using this service will fail
[ 2024.948187] Lustre: Evicted from MGS (at 192.168.204.132@tcp) after server handle changed from 0x14f0a157ec29f0bf to 0x14f0a157ec2a0843
[ 2026.096910] Lustre: 15936:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.132@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2131.166428] LustreError: lustre-MDT0000-mdc-ffff965ac668a800: operation ldlm_enqueue to node 192.168.204.132@tcp failed: rc = -19
[ 2131.197750] LustreError: Skipped 1 previous similar message
[ 2150.898258] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 192.168.204.132@tcp) was lost; in progress operations using this service will fail
[ 2150.938718] Lustre: Evicted from MGS (at 192.168.204.132@tcp) after server handle changed from 0x14f0a157ec2a0843 to 0x14f0a157ec2c36ad
[ 2161.588836] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 16:32:16 (1787603536)
[ 2161.956349] Lustre: *** cfs_fail_loc=305, val=0***
[ 2161.958471] Lustre: Skipped 2 previous similar messages
[ 2202.081228] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 192.168.204.132@tcp) was lost; in progress operations using this service will fail
[ 2212.332163] Lustre: lustre-MDT0000-mdc-ffff965ac668a800: Connection to lustre-MDT0000 (at 192.168.204.132@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2212.334482] Lustre: Evicted from MGS (at 192.168.204.132@tcp) after server handle changed from 0x14f0a157ec2c36ad to 0x14f0a157ec2c5960
[ 2212.349289] Lustre: Skipped 12 previous similar messages
[ 2212.368027] Lustre: MGC192.168.204.132@tcp: Connection restored to 192.168.204.132@tcp (at 192.168.204.132@tcp)
[ 2212.377543] Lustre: Skipped 14 previous similar messages
[ 2227.582588] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2229.438669] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2237.541404] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 16:33:32 (1787603612)
[ 2253.296036] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 192.168.204.132@tcp) was lost; in progress operations using this service will fail
[ 2253.321028] Lustre: Evicted from MGS (at 192.168.204.132@tcp) after server handle changed from 0x14f0a157ec2c5960 to 0x14f0a157ec2c5f8e
[ 2253.369278] LustreError: lustre-MDT0000-mdc-ffff965ac668a800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2270.395843] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 16:34:05 (1787603645)
[ 2284.012410] LustreError: lustre-OST0000-osc-ffff965ac668a800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2299.181724] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 16:34:34 (1787603674)
[ 2312.339613] LustreError: lustre-MDT0000-mdc-ffff965ac668a800: operation ldlm_enqueue to node 192.168.204.132@tcp failed: rc = -19
[ 2330.084815] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 192.168.204.132@tcp) was lost; in progress operations using this service will fail
[ 2340.347235] Lustre: Evicted from MGS (at 192.168.204.132@tcp) after server handle changed from 0x14f0a157ec2c5f8e to 0x14f0a157ec2cb24c
[ 2358.574091] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2360.671969] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2446.835287] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 192.168.204.132@tcp) was lost; in progress operations using this service will fail
[ 2446.876246] Lustre: Evicted from MGS (at 192.168.204.132@tcp) after server handle changed from 0x14f0a157ec2cb24c to 0x14f0a157ec2e7cd3
[ 2448.318224] Lustre: 15936:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.132@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2474.462698] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2476.509183] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2541.756387] LustreError: lustre-MDT0000-mdc-ffff965ac668a800: operation ldlm_enqueue to node 192.168.204.132@tcp failed: rc = -19
[ 2541.767030] LustreError: Skipped 1 previous similar message
[ 2560.480286] Lustre: 2356:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787603920/real 1787603920]  req@ffff965ad8458700 x1874435956653568/t0(0) o400->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 224/224 e 0 to 1 dl 1787603936 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2560.519408] Lustre: 2356:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 15 previous similar messages
[ 2560.523985] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 192.168.204.132@tcp) was lost; in progress operations using this service will fail
[ 2560.569259] Lustre: Evicted from MGS (at 192.168.204.132@tcp) after server handle changed from 0x14f0a157ec2e7cd3 to 0x14f0a157ec306513
[ 2562.283434] Lustre: 15936:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.132@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2586.131561] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2588.275535] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2617.369480] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 16:39:52 (1787603992)
[ 2642.338975] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 192.168.204.132@tcp) was lost; in progress operations using this service will fail
[ 2642.361462] Lustre: Evicted from MGS (at 192.168.204.132@tcp) after server handle changed from 0x14f0a157ec306513 to 0x14f0a157ec312681
[ 2650.497326] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2692.833512] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2728.301602] Lustre: 15936:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.132@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2738.509747] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2778.618581] Lustre: Evicted from MGS (at 192.168.204.132@tcp) after server handle changed from 0x14f0a157ec3179ee to 0x14f0a157ec31b66a
[ 2778.635849] Lustre: Skipped 2 previous similar messages
[ 2788.230317] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2810.840609] LustreError: lustre-MDT0000-mdc-ffff965ac668a800: operation ldlm_enqueue to node 192.168.204.132@tcp failed: rc = -19
[ 2810.854637] LustreError: Skipped 5 previous similar messages
[ 2828.768256] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 192.168.204.132@tcp) was lost; in progress operations using this service will fail
[ 2828.776857] LustreError: Skipped 3 previous similar messages
[ 2828.800214] Lustre: MGC192.168.204.132@tcp: Connection restored to 192.168.204.132@tcp (at 192.168.204.132@tcp)
[ 2828.813150] Lustre: Skipped 16 previous similar messages
[ 2836.268130] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2864.239672] Lustre: lustre-MDT0000-mdc-ffff965ac668a800: Connection to lustre-MDT0000 (at 192.168.204.132@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2864.260891] Lustre: Skipped 9 previous similar messages
[ 2900.266904] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2998.421919] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 16:46:13 (1787604373)
[ 3056.054715] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3058.470833] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3345.562669] LustreError: lustre-OST0000-osc-ffff965ac668a800: operation ldlm_enqueue to node 192.168.204.132@tcp failed: rc = -19
[ 3345.574511] LustreError: Skipped 4 previous similar messages
[ 3390.248234] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3392.640369] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3678.196906] Lustre: lustre-OST0000-osc-ffff965ac668a800: Connection to lustre-OST0000 (at 192.168.204.132@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3678.211014] Lustre: Skipped 3 previous similar messages
[ 3716.029822] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3718.063519] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3975.522847] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 17:02:29 (1787605349)
[ 3993.568274] Lustre: 51019:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787605353/real 1787605353]  req@ffff965ac5778380 x1874435975555584/t0(0) o101->lustre-MDT0000-mdc-ffff965ac668a800@192.168.204.132@tcp:12/10 lens 576/1152 e 0 to 1 dl 1787605369 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3993.602902] Lustre: 51019:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
[ 3993.642683] Lustre: lustre-MDT0000-mdc-ffff965ac668a800: Connection restored to 192.168.204.132@tcp (at 192.168.204.132@tcp)
[ 3993.661206] Lustre: Skipped 6 previous similar messages
[ 4001.943782] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 17:02:56 (1787605376)
[ 4029.170169] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 17:03:23 (1787605403)
[ 4056.800176] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 17:03:51 (1787605431)
[ 4057.341276] Lustre: Mounted lustre-client - version 2.17.57_82_gfc0532e
[ 4089.312597] Lustre: 2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787605449/real 1787605449]  req@ffff965ad85af800 x1874435975572864/t0(0) o400->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 224/224 e 0 to 1 dl 1787605465 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4089.355017] Lustre: 2357:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
[ 4089.362834] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 192.168.204.132@tcp) was lost; in progress operations using this service will fail
[ 4089.373051] LustreError: Skipped 2 previous similar messages
[ 4099.580434] Lustre: Evicted from MGS (at 192.168.204.132@tcp) after server handle changed from 0x14f0a157ec33417a to 0x14f0a157ec4696ff
[ 4099.591483] Lustre: Skipped 3 previous similar messages
[ 4106.676886] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4108.299108] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4110.549155] Lustre: Unmounted lustre-client
[ 4117.420425] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 17:04:52 (1787605492)
[ 4239.840170] Lustre: 2356:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787605599/real 1787605599]  req@ffff965ac5778000 x1874435975606400/t0(0) o4->lustre-OST0000-osc-ffff965ac668a800@192.168.204.132@tcp:6/4 lens 488/448 e 0 to 1 dl 1787605615 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4239.910890] Lustre: 2356:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 61 previous similar messages
[ 4287.971954] Lustre: lustre-OST0000-osc-ffff965ac668a800: Connection to lustre-OST0000 (at 192.168.204.132@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4287.992745] Lustre: Skipped 14 previous similar messages
[ 4418.680898] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 17:09:52 (1787605792)
[ 4471.619612] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 17:10:45 (1787605845)
[ 4475.700892] Lustre: Unmounted lustre-client
[ 4499.621940] Lustre: Mounted lustre-client - version 2.17.57_82_gfc0532e
[ 4507.989360] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 17:11:22 (1787605882)
[ 4508.337950] LustreError: 56948:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4509.402114] LustreError: 56948:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4509.699670] Lustre: *** cfs_fail_loc=305, val=0***
[ 4534.214639] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 17:11:48 (1787605908)
[ 4534.789723] Lustre: Mounted lustre-client - version 2.17.57_82_gfc0532e
[ 4538.150500] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4548.442668] Lustre: Unmounted lustre-client
[ 4556.643613] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 17:12:11 (1787605931)
[ 4660.121769] LustreError: lustre-MDT0000-mdc-ffff965ac78cd800: operation mds_reint to node 192.168.204.132@tcp failed: rc = -19
[ 4660.133479] LustreError: Skipped 2 previous similar messages
[ 4679.145263] Lustre: 2356:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787606039/real 1787606039]  req@ffff965ac628ad80 x1874435978280960/t0(0) o400->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 224/224 e 0 to 1 dl 1787606055 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4679.201614] Lustre: 2356:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 72 previous similar messages
[ 4679.215783] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 192.168.204.132@tcp) was lost; in progress operations using this service will fail
[ 4689.445099] Lustre: Evicted from MGS (at 192.168.204.132@tcp) after server handle changed from 0x14f0a157ec46ad41 to 0x14f0a157ec49611b
[ 4689.476762] Lustre: MGC192.168.204.132@tcp: Connection restored to 192.168.204.132@tcp (at 192.168.204.132@tcp)
[ 4689.489506] Lustre: Skipped 23 previous similar messages
[ 4880.969496] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 17:17:35 (1787606255)
[ 4888.908950] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4903.915717] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 192.168.204.132@tcp) was lost; in progress operations using this service will fail
[ 4903.936734] Lustre: lustre-MDT0000-mdc-ffff965ac78cd800: Connection to lustre-MDT0000 (at 192.168.204.132@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4903.974880] Lustre: Skipped 10 previous similar messages
[ 4903.983458] Lustre: Evicted from MGS (at 192.168.204.132@tcp) after server handle changed from 0x14f0a157ec49611b to 0x14f0a157ec4e2c28
[ 4904.182915] LustreError: lustre-MDT0000-mdc-ffff965ac78cd800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4922.848432] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 17:18:17 (1787606297)
[ 4923.447574] Lustre: Mounted lustre-client - version 2.17.57_82_gfc0532e
[ 4931.054909] LustreError: lustre-OST0000-osc-ffff965ac78cd800: operation ldlm_enqueue to node 192.168.204.132@tcp failed: rc = -107
[ 4931.074537] LustreError: lustre-OST0000-osc-ffff965ac78cd800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4931.088576] LustreError: 60676:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff965ac78cd800: namespace resource [0x240000400:0x3be3:0x0].0x0 (ffff965ac6c01400) refcount nonzero (2) after lock cleanup; forcing cleanup.
[ 4944.764144] Lustre: Unmounted lustre-client
[ 4953.080035] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 17:18:47 (1787606327)
[ 4960.176981] LustreError: lustre-MDT0000-mdc-ffff965ac78cd800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4960.193976] LustreError: 61432:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff965ac78cd800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff965ac638c200) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4960.196396] LustreError: 61348:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4967.840393] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 17:19:02 (1787606342)
[ 4968.200275] LustreError: 62052:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4972.448744] LustreError: lustre-MDT0000-mdc-ffff965ac78cd800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4972.467857] LustreError: 62069:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4972.480795] LustreError: 62052:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=726
[ 4972.502327] LustreError: 62052:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting
[ 4972.569817] LustreError: 62074:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4973.718060] LustreError: 62080:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4973.729944] LustreError: 62080:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4976.053259] LustreError: 62102:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4976.068304] LustreError: 62102:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4980.774946] LustreError: 62147:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4980.794714] LustreError: 62147:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4992.653950] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 17:19:26 (1787606366)
[ 5039.231675] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5041.017484] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5053.553897] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 17:20:27 (1787606427)
[ 5095.092679] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5097.382530] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5111.447633] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 17:21:25 (1787606485)
[ 5183.150492] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5185.334438] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5195.962211] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 17:22:50 (1787606570)
[ 5238.808443] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5240.754262] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5248.332571] Lustre: Unmounted lustre-client
[ 5307.330387] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5309.995029] Lustre: Mounted lustre-client - version 2.17.57_82_gfc0532e
[ 5317.907211] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 17:24:52 (1787606692)
[ 5320.339467] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5321.888288] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 17:24:56 (1787606696)
[ 5354.388193] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 17:25:28 (1787606728)
[ 5356.490900] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5358.951249] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 17:25:33 (1787606733)
[ 5360.747034] Lustre: *** cfs_fail_loc=805, val=0***
[ 5360.810420] Lustre: Mounted lustre-client - version 2.17.57_82_gfc0532e
[ 5366.999951] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5387.744822] Lustre: 2354:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787606747/real 1787606747]  req@ffff965ac9957480 x1874435980341504/t0(0) o400->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 224/224 e 0 to 1 dl 1787606763 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5387.830853] Lustre: 2354:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5387.844592] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 192.168.204.132@tcp) was lost; in progress operations using this service will fail
[ 5398.514533] Lustre: Evicted from MGS (at 192.168.204.132@tcp) after server handle changed from 0x14f0a157ec4e3830 to 0x14f0a157ec4e3ab4
[ 5398.528776] Lustre: MGC192.168.204.132@tcp: Connection restored to 192.168.204.132@tcp (at 192.168.204.132@tcp)
[ 5398.534493] Lustre: Skipped 11 previous similar messages
[ 5410.368604] Lustre: Unmounted lustre-client
[ 5419.653620] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 17:26:34 (1787606794)
[ 5447.168118] Lustre: 68713:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.132@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 5462.062291] LustreError: 2353:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff965ac9954a80 x1874435980354304/t90194313220(90194313220) o101->lustre-MDT0000-mdc-ffff965ac7b6b800@192.168.204.132@tcp:12/10 lens 664/608 e 0 to 0 dl 1787606854 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5473.551334] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5475.114795] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5484.855357] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 17:27:39 (1787606859)
[ 5486.463362] LustreError: lustre-OST0000-osc-ffff965ac7b6b800: operation ost_write to node 192.168.204.132@tcp failed: rc = -107
[ 5486.473174] LustreError: Skipped 2 previous similar messages
[ 5486.487342] LustreError: lustre-OST0000-osc-ffff965ac7b6b800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5486.506563] Lustre: 2356:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.132@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5486.561167] LustreError: 73258:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff965ac7b6b800: namespace resource [0x240000400:0x3c42:0x0].0x0 (ffff965ac6c01100) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5496.175261] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 17:27:50 (1787606870)
[ 5497.896761] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5499.965830] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 17:27:54 (1787606874)
[ 5501.915926] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5503.764506] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 17:27:58 (1787606878)
[ 5505.226740] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5507.252808] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 17:28:02 (1787606882)
[ 5509.111361] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5511.373452] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 17:28:05 (1787606885)
[ 5513.373035] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5515.747652] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 17:28:09 (1787606889)
[ 5517.679126] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5520.138056] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 17:28:14 (1787606894)
[ 5521.640430] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5523.788961] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 17:28:18 (1787606898)
[ 5525.767362] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5527.667250] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 17:28:22 (1787606902)
[ 5530.210638] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5532.789775] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 17:28:27 (1787606907)
[ 5534.993476] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5537.533772] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 17:28:31 (1787606911)
[ 5540.648992] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5543.279594] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 17:28:37 (1787606917)
[ 5546.602379] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5548.355773] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 17:28:43 (1787606923)
[ 5569.509082] Lustre: lustre-MDT0000-mdc-ffff965ac7b6b800: Connection to lustre-MDT0000 (at 192.168.204.132@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5569.537699] Lustre: Skipped 13 previous similar messages
[ 5583.797982] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 17:29:17 (1787606957)
[ 5615.777298] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 17:29:50 (1787606990)
[ 5616.538466] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5627.218205] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 17:30:02 (1787607002)
[ 5627.759636] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5637.606474] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 17:30:12 (1787607012)
[ 5637.972153] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5646.227990] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 17:30:21 (1787607021)
[ 5646.608498] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5654.723617] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 17:30:29 (1787607029)
[ 5655.111082] Lustre: *** cfs_fail_loc=510, val=3***
[ 5663.685127] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 17:30:38 (1787607038)
[ 5663.971994] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5674.082762] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 17:30:48 (1787607048)
[ 5674.395835] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5741.350400] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 17:31:55 (1787607115)
[ 5741.771952] LustreError: 83467:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5744.647378] LustreError: lustre-MDT0000-mdc-ffff965ac7b6b800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5744.672133] LustreError: 83481:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff965ac7b6b800: inode [0x20000a042:0xe:0x0] mdc close failed: rc = -108
[ 5744.701850] LustreError: 83481:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff965ac7b6b800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff965ac62ab500) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5745.840174] LustreError: 83467:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5747.986897] LustreError: 83497:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5750.912675] LustreError: lustre-MDT0000-mdc-ffff965ac7b6b800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5750.938257] LustreError: 83511:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff965ac7b6b800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff965ac62ab500) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5750.972041] LustreError: 83511:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5752.025228] LustreError: 83497:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5759.122787] LustreError: lustre-MDT0000-mdc-ffff965ac7b6b800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5759.672886] LustreError: 83552:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5762.829031] LustreError: lustre-MDT0000-mdc-ffff965ac7b6b800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5762.934599] LustreError: 83572:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5762.944783] LustreError: 83572:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5763.760230] LustreError: 83552:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5763.778066] LustreError: 83552:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff965ac7b6b800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5763.794450] LustreError: 83552:0:(file.c:251:ll_close_inode_openhandle()) Skipped 1 previous similar message
[ 5764.057659] LustreError: 83578:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5764.079341] LustreError: 83578:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5766.393239] LustreError: 83600:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5766.401602] LustreError: 83600:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5770.107150] LustreError: 83627:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5772.989089] LustreError: lustre-MDT0000-mdc-ffff965ac7b6b800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5773.015586] LustreError: 83641:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff965ac7b6b800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff965ac5f53a00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5773.048196] LustreError: 83641:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5774.125357] LustreError: 83627:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5781.584505] LustreError: lustre-MDT0000-mdc-ffff965ac7b6b800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5782.142306] LustreError: 83682:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5785.190056] LustreError: lustre-MDT0000-mdc-ffff965ac7b6b800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5786.224777] LustreError: 83682:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5789.713526] LustreError: lustre-MDT0000-mdc-ffff965ac7b6b800: operation ldlm_enqueue to node 192.168.204.132@tcp failed: rc = -107
[ 5789.725432] LustreError: Skipped 7 previous similar messages
[ 5789.753099] LustreError: lustre-MDT0000-mdc-ffff965ac7b6b800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5789.870206] LustreError: 83730:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5789.904180] LustreError: 83730:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5790.497757] LustreError: 83710:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff965ac7b6b800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5793.831139] LustreError: 83725:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff965ac7b6b800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff965ac62ab200) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5793.863678] LustreError: 83725:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 3 previous similar messages
[ 5799.388395] LustreError: 83781:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5799.413718] LustreError: 83781:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5803.226351] LustreError: lustre-MDT0000-mdc-ffff965ac7b6b800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5803.308945] LustreError: 83804:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5803.323894] LustreError: 83804:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 24 previous similar messages
[ 5803.406416] LustreError: 83781:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5803.420031] LustreError: 83781:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5803.428276] LustreError: 83781:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff965ac7b6b800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5814.943647] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 17:33:09 (1787607189)
[ 5834.270928] LustreError: 68710:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5834.296397] LustreError: 68710:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5838.352183] LustreError: 68710:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5838.370101] LustreError: 68710:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5846.930117] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 17:33:40 (1787607220)
[ 5898.989827] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 17:34:33 (1787607273)
[ 5965.120760] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 17:35:39 (1787607339)
[ 5999.308820] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 17:36:14 (1787607374)
[ 5999.907972] Lustre: Mounted lustre-client - version 2.17.57_82_gfc0532e
[ 6124.293549] Lustre: Unmounted lustre-client
[ 6131.430777] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 17:38:26 (1787607506)
[ 6132.133480] LustreError: 88106:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 6134.634246] LustreError: lustre-OST0000-osc-ffff965ac7b6b800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6134.657436] LustreError: 88188:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff965ac7b6b800: namespace resource [0x240000400:0x3c6d:0x0].0x0 (ffff965ac5f53800) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6135.064988] LustreError: 88106:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout interrupted
[ 6135.080616] Lustre: 2354:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.132@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 6142.076263] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 17:38:36 (1787607516)
[ 6200.288254] Lustre: 88773:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787607521/real 1787607521]  req@ffff965ad0ae4a80 x1874435980530048/t0(0) o101->lustre-MDT0000-mdc-ffff965ac7b6b800@192.168.204.132@tcp:12/10 lens 328/344 e 0 to 1 dl 1787607576 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 6200.342917] Lustre: 88773:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 22 previous similar messages
[ 6200.361601] Lustre: lustre-MDT0000-mdc-ffff965ac7b6b800: Connection to lustre-MDT0000 (at 192.168.204.132@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6200.377133] Lustre: Skipped 14 previous similar messages
[ 6200.405107] Lustre: lustre-MDT0000-mdc-ffff965ac7b6b800: Connection restored to 192.168.204.132@tcp (at 192.168.204.132@tcp)
[ 6200.421260] Lustre: Skipped 18 previous similar messages
[ 6208.495295] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 17:39:42 (1787607582)
[ 6210.147363] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 6212.318737] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 17:39:46 (1787607586)
[ 6275.347364] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6277.013328] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 17:40:51 (1787607651)
[ 6335.327539] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 17:41:49 (1787607709)
[ 6337.818646] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6339.744637] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 17:41:54 (1787607714)
[ 6341.971307] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6344.247421] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 17:41:58 (1787607718)
[ 6379.397538] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 17:42:34 (1787607754)
[ 6392.182790] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6416.352537] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 192.168.204.132@tcp) was lost; in progress operations using this service will fail
[ 6416.373283] LustreError: Skipped 2 previous similar messages
[ 6426.668361] Lustre: Evicted from MGS (at 192.168.204.132@tcp) after server handle changed from 0x14f0a157ec4e4684 to 0x14f0a157ec4e5fa5
[ 6426.686016] Lustre: Skipped 2 previous similar messages
[ 6440.095843] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6441.595136] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6451.882191] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 17:43:46 (1787607826)
[ 6454.301631] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6456.155771] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 17:43:50 (1787607830)
[ 6468.076317] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 192.168.204.132@tcp) was lost; in progress operations using this service will fail
[ 6468.109727] Lustre: Evicted from MGS (at 192.168.204.132@tcp) after server handle changed from 0x14f0a157ec4e5fa5 to 0x14f0a157ec4e62df
[ 6483.998899] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 17:44:18 (1787607858)
[ 6538.328315] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 17:45:13 (1787607913)
[ 6546.369362] LustreError: lustre-OST0000-osc-ffff965ac7b6b800: operation ost_setattr to node 192.168.204.132@tcp failed: rc = -107
[ 6546.397487] LustreError: Skipped 2 previous similar messages
[ 6596.906557] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6600.338724] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6683.620825] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 192.168.204.132@tcp) was lost; in progress operations using this service will fail
[ 6683.647429] LustreError: Skipped 1 previous similar message
[ 6692.857047] Lustre: Evicted from MGS (at 192.168.204.132@tcp) after server handle changed from 0x14f0a157ec4e656a to 0x14f0a157ec4e73b7
[ 6692.893213] Lustre: Skipped 1 previous similar message
[ 6694.202120] Lustre: 68713:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.132@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6711.273580] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6712.758710] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6754.692410] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6756.380243] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6897.111945] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 17:51:12 (1787608272)
[ 6907.426472] Lustre: lustre-OST0000-osc-ffff965ac7b6b800: Connection to lustre-OST0000 (at 192.168.204.132@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6907.448662] Lustre: Skipped 8 previous similar messages
[ 6962.198048] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6965.759467] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 7223.558241] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 17:56:38 (1787608598)
[ 7379.941743] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 192.168.204.132@tcp) was lost; in progress operations using this service will fail
[ 7379.952792] LustreError: Skipped 1 previous similar message
[ 7379.972238] Lustre: Evicted from MGS (at 192.168.204.132@tcp) after server handle changed from 0x14f0a157ec4e7a40 to 0x14f0a157ec5e1ae3
[ 7379.987122] Lustre: Skipped 1 previous similar message
[ 7380.000373] Lustre: MGC192.168.204.132@tcp: Connection restored to 192.168.204.132@tcp (at 192.168.204.132@tcp)
[ 7380.009769] Lustre: Skipped 12 previous similar messages
[ 7415.356447] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 17:59:49 (1787608789)
[ 7417.196412] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7419.123812] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 17:59:53 (1787608793)
[ 7421.240053] LustreError: lustre-MDT0000-mdc-ffff965ac7b6b800: operation ldlm_enqueue to node 192.168.204.132@tcp failed: rc = -107
[ 7421.251399] LustreError: Skipped 229 previous similar messages
[ 7421.262319] LustreError: lustre-MDT0000-mdc-ffff965ac7b6b800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7429.794063] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 18:00:04 (1787608804)
[ 7597.055232] LustreError: lustre-OST0000-osc-ffff965ac7b6b800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7600.264497] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 18:02:53 (1787608973)
[ 7627.746688] Lustre: 2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787608983/real 1787608983]  req@ffff965aee23d500 x1874435998263168/t0(0) o4->lustre-OST0000-osc-ffff965ac7b6b800@192.168.204.132@tcp:6/4 lens 4584/448 e 0 to 1 dl 1787609003 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7627.781446] Lustre: 2355:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 29 previous similar messages
[ 7627.802213] Lustre: lustre-OST0000-osc-ffff965ac7b6b800: Connection to lustre-OST0000 (at 192.168.204.132@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7627.827388] Lustre: Skipped 3 previous similar messages
[ 7644.558714] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 18:03:39 (1787609019)
[ 7645.928924] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7647.837502] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 18:03:42 (1787609022)
[ 7649.238364] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7651.002991] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 18:03:45 (1787609025)
[ 7686.293644] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 18:04:20 (1787609060)
[ 7725.045782] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 192.168.204.132@tcp) was lost; in progress operations using this service will fail
[ 7725.089553] Lustre: Evicted from MGS (at 192.168.204.132@tcp) after server handle changed from 0x14f0a157ec5e1ae3 to 0x14f0a157ec5e41d3
[ 7725.095991] Lustre: Skipped 1 previous similar message
[ 7741.768808] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 18:05:16 (1787609116)
[ 7743.503736] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7745.668657] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 18:05:20 (1787609120)
[ 7747.415712] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7749.550516] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 18:05:24 (1787609124)
[ 7752.288924] Lustre: 2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787609090/real 1787609090]  req@ffff965ac63ac000 x1874435998418688/t0(0) o400->lustre-MDT0000-mdc-ffff965ac7b6b800@192.168.204.132@tcp:12/10 lens 224/224 e 0 to 1 dl 1787609127 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7761.513394] Lustre: Unmounted lustre-client
[ 7763.810879] Lustre: Mounted lustre-client - version 2.17.57_82_gfc0532e
[ 7766.999750] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7809.617131] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2