[ 1840.057252] LustreError: lustre-OST0000-osc-ffff8c979843b800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1840.068656] Lustre: 2358:0:(llite_lib.c:4327:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.150@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1846.732705] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 02:38:59 (1787035139)
[ 1848.795312] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1850.411214] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 02:39:03 (1787035143)
[ 1852.092789] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1853.324314] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 02:39:06 (1787035146)
[ 1857.100889] LustreError: lustre-MDT0000-mdc-ffff8c979843b800: operation mds_close to node 192.168.201.150@tcp failed: rc = -19
[ 1857.117967] LustreError: Skipped 2 previous similar messages
[ 1876.964706] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 192.168.201.150@tcp) was lost; in progress operations using this service will fail
[ 1887.244483] Lustre: Evicted from MGS (at 192.168.201.150@tcp) after server handle changed from 0x437ef3b07a71d216 to 0x437ef3b07a71ec09
[ 1990.085897] LustreError: lustre-MDT0000-mdc-ffff8c979843b800: operation ldlm_enqueue to node 192.168.201.150@tcp failed: rc = -19
[ 2009.084306] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 192.168.201.150@tcp) was lost; in progress operations using this service will fail
[ 2009.119902] Lustre: Evicted from MGS (at 192.168.201.150@tcp) after server handle changed from 0x437ef3b07a71ec09 to 0x437ef3b07a740668
[ 2020.708426] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 02:41:54 (1787035314)
[ 2021.034547] Lustre: *** cfs_fail_loc=305, val=0***
[ 2021.037504] Lustre: Skipped 2 previous similar messages
[ 2061.282188] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 192.168.201.150@tcp) was lost; in progress operations using this service will fail
[ 2070.589481] Lustre: Evicted from MGS (at 192.168.201.150@tcp) after server handle changed from 0x437ef3b07a740668 to 0x437ef3b07a742af7
[ 2080.421858] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2082.067848] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2090.888288] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 02:43:03 (1787035383)
[ 2106.347511] Lustre: lustre-MDT0000-mdc-ffff8c979843b800: Connection to lustre-MDT0000 (at 192.168.201.150@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2106.359225] Lustre: Skipped 12 previous similar messages
[ 2106.366093] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 192.168.201.150@tcp) was lost; in progress operations using this service will fail
[ 2106.418639] Lustre: Evicted from MGS (at 192.168.201.150@tcp) after server handle changed from 0x437ef3b07a742af7 to 0x437ef3b07a742da5
[ 2106.448786] LustreError: lustre-MDT0000-mdc-ffff8c979843b800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2106.451889] Lustre: MGC192.168.201.150@tcp: Connection restored to 192.168.201.150@tcp (at 192.168.201.150@tcp)
[ 2106.471286] Lustre: Skipped 16 previous similar messages
[ 2127.068811] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 02:43:39 (1787035419)
[ 2147.321827] LustreError: lustre-OST0000-osc-ffff8c979843b800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2168.616476] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 02:44:21 (1787035461)
[ 2183.036827] LustreError: lustre-MDT0000-mdc-ffff8c979843b800: operation ldlm_enqueue to node 192.168.201.150@tcp failed: rc = -19
[ 2203.551200] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 192.168.201.150@tcp) was lost; in progress operations using this service will fail
[ 2203.608221] Lustre: Evicted from MGS (at 192.168.201.150@tcp) after server handle changed from 0x437ef3b07a742da5 to 0x437ef3b07a746ed5
[ 2233.312176] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2236.512863] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2321.375177] Lustre: 2356:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787035599/real 1787035599]  req@ffff8c978847fb80 x1873840315459584/t0(0) o400->MGC192.168.201.150@tcp@192.168.201.150@tcp:26/25 lens 224/224 e 0 to 1 dl 1787035615 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2321.418053] Lustre: 2356:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 10 previous similar messages
[ 2321.427772] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 192.168.201.150@tcp) was lost; in progress operations using this service will fail
[ 2330.613508] Lustre: Evicted from MGS (at 192.168.201.150@tcp) after server handle changed from 0x437ef3b07a746ed5 to 0x437ef3b07a763087
[ 2351.502724] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2353.808977] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2420.147420] LustreError: lustre-MDT0000-mdc-ffff8c979843b800: operation ldlm_enqueue to node 192.168.201.150@tcp failed: rc = -19
[ 2420.163069] LustreError: Skipped 2 previous similar messages
[ 2438.111228] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 192.168.201.150@tcp) was lost; in progress operations using this service will fail
[ 2448.308805] Lustre: Evicted from MGS (at 192.168.201.150@tcp) after server handle changed from 0x437ef3b07a763087 to 0x437ef3b07a77f389
[ 2464.670289] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2466.817073] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2496.561992] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 02:49:49 (1787035789)
[ 2519.008862] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 192.168.201.150@tcp) was lost; in progress operations using this service will fail
[ 2528.331655] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2529.218767] Lustre: Evicted from MGS (at 192.168.201.150@tcp) after server handle changed from 0x437ef3b07a77f389 to 0x437ef3b07a78c2c6
[ 2558.211239] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2593.976853] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2637.612851] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2681.331741] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 192.168.201.150@tcp) was lost; in progress operations using this service will fail
[ 2681.341984] LustreError: Skipped 3 previous similar messages
[ 2681.363958] Lustre: Evicted from MGS (at 192.168.201.150@tcp) after server handle changed from 0x437ef3b07a78e24d to 0x437ef3b07a793c20
[ 2681.385859] Lustre: Skipped 3 previous similar messages
[ 2683.382071] Lustre: 15883:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.150@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2699.434702] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2727.752633] LustreError: lustre-MDT0000-mdc-ffff8c979843b800: operation ldlm_enqueue to node 192.168.201.150@tcp failed: rc = -19
[ 2727.767192] LustreError: Skipped 9 previous similar messages
[ 2727.773100] Lustre: lustre-MDT0000-mdc-ffff8c979843b800: Connection to lustre-MDT0000 (at 192.168.201.150@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2727.801104] Lustre: Skipped 8 previous similar messages
[ 2758.068369] Lustre: MGC192.168.201.150@tcp: Connection restored to 192.168.201.150@tcp (at 192.168.201.150@tcp)
[ 2758.079936] Lustre: Skipped 16 previous similar messages
[ 2764.864154] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2853.331493] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 02:55:46 (1787036146)
[ 2905.906822] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 2908.165239] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3244.347023] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3246.735525] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3531.774406] LustreError: lustre-OST0000-osc-ffff8c979843b800: operation ldlm_enqueue to node 192.168.201.150@tcp failed: rc = -19
[ 3531.793099] LustreError: Skipped 4 previous similar messages
[ 3531.805977] Lustre: lustre-OST0000-osc-ffff8c979843b800: Connection to lustre-OST0000 (at 192.168.201.150@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3531.831842] Lustre: Skipped 3 previous similar messages
[ 3553.246586] Lustre: lustre-OST0000-osc-ffff8c979843b800: Connection restored to 192.168.201.150@tcp (at 192.168.201.150@tcp)
[ 3553.260814] Lustre: Skipped 5 previous similar messages
[ 3572.356247] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3575.143258] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3826.830616] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 03:11:59 (1787037119)
[ 3845.087203] Lustre: 50380:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787037123/real 1787037123]  req@ffff8c979001d500 x1873840337259776/t0(0) o101->lustre-MDT0000-mdc-ffff8c979843b800@192.168.201.150@tcp:12/10 lens 576/1152 e 0 to 1 dl 1787037139 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3845.155624] Lustre: 50380:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
[ 3855.217765] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 03:12:28 (1787037148)
[ 3880.736704] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 03:12:53 (1787037173)
[ 3907.573086] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 03:13:20 (1787037200)
[ 3908.324893] Lustre: Mounted lustre-client
[ 3940.255211] Lustre: 2359:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787037218/real 1787037218]  req@ffff8c9788ce4700 x1873840337278592/t0(0) o400->lustre-MDT0000-mdc-ffff8c979843b800@192.168.201.150@tcp:12/10 lens 224/224 e 0 to 1 dl 1787037234 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3940.302056] Lustre: 2359:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 3940.316363] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 192.168.201.150@tcp) was lost; in progress operations using this service will fail
[ 3940.326989] LustreError: Skipped 2 previous similar messages
[ 3949.581498] Lustre: Evicted from MGS (at 192.168.201.150@tcp) after server handle changed from 0x437ef3b07a7a95ff to 0x437ef3b07a8fa9cb
[ 3949.599920] Lustre: Skipped 2 previous similar messages
[ 3958.669731] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 3960.324960] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3962.549543] Lustre: Unmounted lustre-client
[ 3970.060648] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 03:14:22 (1787037262)
[ 4093.215291] Lustre: 2359:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787037371/real 1787037371]  req@ffff8c9789768000 x1873840337312640/t0(0) o4->lustre-OST0000-osc-ffff8c979843b800@192.168.201.150@tcp:6/4 lens 488/448 e 0 to 1 dl 1787037387 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4093.244957] Lustre: 2359:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 58 previous similar messages
[ 4140.511236] Lustre: lustre-OST0000-osc-ffff8c979843b800: Connection to lustre-OST0000 (at 192.168.201.150@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4140.530347] Lustre: Skipped 14 previous similar messages
[ 4156.747203] Lustre: lustre-OST0000-osc-ffff8c979843b800: Connection restored to 192.168.201.150@tcp (at 192.168.201.150@tcp)
[ 4156.769423] Lustre: Skipped 16 previous similar messages
[ 4268.594160] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 03:19:21 (1787037561)
[ 4319.175789] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 03:20:11 (1787037611)
[ 4323.249060] Lustre: Unmounted lustre-client
[ 4348.388053] Lustre: Mounted lustre-client
[ 4355.916833] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 03:20:48 (1787037648)
[ 4356.155366] LustreError: 56293:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4357.208498] LustreError: 56293:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4357.432589] Lustre: *** cfs_fail_loc=305, val=0***
[ 4379.849439] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 03:21:13 (1787037673)
[ 4380.233937] Lustre: Mounted lustre-client
[ 4382.296121] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4392.638213] Lustre: Unmounted lustre-client
[ 4401.297921] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 03:21:34 (1787037694)
[ 4485.657123] LustreError: lustre-MDT0000-mdc-ffff8c97883ac000: operation mds_reint to node 192.168.201.150@tcp failed: rc = -19
[ 4504.031516] Lustre: 2357:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787037782/real 1787037782]  req@ffff8c97898fed80 x1873840339981696/t0(0) o400->MGC192.168.201.150@tcp@192.168.201.150@tcp:26/25 lens 224/224 e 0 to 1 dl 1787037798 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4504.047891] Lustre: 2357:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 83 previous similar messages
[ 4504.053946] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 192.168.201.150@tcp) was lost; in progress operations using this service will fail
[ 4514.211558] Lustre: Evicted from MGS (at 192.168.201.150@tcp) after server handle changed from 0x437ef3b07a8fca78 to 0x437ef3b07a9281e7
[ 4678.395351] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 03:26:11 (1787037971)
[ 4683.771662] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4697.572077] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 192.168.201.150@tcp) was lost; in progress operations using this service will fail
[ 4697.604564] Lustre: Evicted from MGS (at 192.168.201.150@tcp) after server handle changed from 0x437ef3b07a9281e7 to 0x437ef3b07a974855
[ 4706.913847] LustreError: lustre-MDT0000-mdc-ffff8c97883ac000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4717.610275] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 03:26:50 (1787038010)
[ 4718.400946] Lustre: Mounted lustre-client
[ 4726.462655] LustreError: lustre-OST0000-osc-ffff8c97883ac000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4726.484647] LustreError: 60006:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff8c97883ac000: namespace resource [0x240000400:0x4043:0x0].0x0 (ffff8c9789857f00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4739.015302] Lustre: Unmounted lustre-client
[ 4746.512594] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 03:27:19 (1787038039)
[ 4750.141291] Lustre: lustre-MDT0000-mdc-ffff8c97883ac000: Connection to lustre-MDT0000 (at 192.168.201.150@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4750.151181] Lustre: Skipped 14 previous similar messages
[ 4753.127653] LustreError: lustre-MDT0000-mdc-ffff8c97883ac000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4753.149539] LustreError: 60678:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4761.641830] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 03:27:34 (1787038054)
[ 4761.879963] LustreError: 61378:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4765.669885] Lustre: lustre-MDT0000-mdc-ffff8c97883ac000: Connection restored to 192.168.201.150@tcp (at 192.168.201.150@tcp)
[ 4765.688551] Lustre: Skipped 16 previous similar messages
[ 4767.199137] LustreError: 61378:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=0
[ 4767.204777] LustreError: 61378:0:(import.c:711:ptlrpc_connect_import_locked()) already connected
[ 4767.672316] LustreError: lustre-MDT0000-mdc-ffff8c97883ac000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4767.686052] LustreError: 61399:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4767.762065] LustreError: 61404:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4768.885215] LustreError: 61411:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4768.898981] LustreError: 61411:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4771.325352] LustreError: 61433:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4771.346737] LustreError: 61433:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4776.236374] LustreError: 61478:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4776.247191] LustreError: 61478:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4785.649514] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 03:27:58 (1787038078)
[ 4831.939456] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4834.353042] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4847.428533] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 03:29:00 (1787038140)
[ 4889.446539] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4891.144525] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4902.760421] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 03:29:55 (1787038195)
[ 4968.965668] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4970.685981] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4980.626746] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 03:31:13 (1787038273)
[ 5022.409719] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5024.672420] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5031.188807] Lustre: Unmounted lustre-client
[ 5086.492421] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5089.206756] Lustre: Mounted lustre-client
[ 5096.583413] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 03:33:10 (1787038390)
[ 5099.371922] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5101.181355] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 03:33:14 (1787038394)
[ 5136.637477] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 03:33:49 (1787038429)
[ 5138.819831] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5140.407527] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 03:33:53 (1787038433)
[ 5141.750780] Lustre: *** cfs_fail_loc=805, val=0***
[ 5141.820614] Lustre: Mounted lustre-client
[ 5147.808590] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5167.583152] Lustre: 2357:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787038446/real 1787038446]  req@ffff8c9784685180 x1873840342033024/t0(0) o400->lustre-MDT0000-mdc-ffff8c9788fdd000@192.168.201.150@tcp:12/10 lens 224/224 e 0 to 1 dl 1787038462 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5167.634029] Lustre: 2357:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5167.647801] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 192.168.201.150@tcp) was lost; in progress operations using this service will fail
[ 5177.830829] Lustre: Evicted from MGS (at 192.168.201.150@tcp) after server handle changed from 0x437ef3b07a9752f8 to 0x437ef3b07a975513
[ 5193.194691] LustreError: 2355:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8c9785e8e000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff8c97898c7900) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5196.133812] Lustre: Unmounted lustre-client
[ 5205.992900] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 03:34:58 (1787038498)
[ 5232.695553] LustreError: 2355:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff8c9784686a00 x1873840342044672/t90194313220(90194313220) o101->lustre-MDT0000-mdc-ffff8c9788fdd000@192.168.201.150@tcp:12/10 lens 664/608 e 0 to 0 dl 1787038543 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5243.875876] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5246.255940] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5256.902992] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 03:35:49 (1787038549)
[ 5258.449362] LustreError: lustre-OST0000-osc-ffff8c9788fdd000: operation ost_write to node 192.168.201.150@tcp failed: rc = -107
[ 5258.459692] LustreError: Skipped 3 previous similar messages
[ 5258.471112] LustreError: lustre-OST0000-osc-ffff8c9788fdd000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5258.481573] Lustre: 2358:0:(llite_lib.c:4327:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.150@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5258.498252] LustreError: 72520:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff8c9788fdd000: namespace resource [0x240000400:0x40a2:0x0].0x0 (ffff8c978990f800) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5258.509656] LustreError: 72520:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5266.129874] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 03:35:59 (1787038559)
[ 5267.640673] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5269.558553] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 03:36:02 (1787038562)
[ 5271.097583] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5273.072369] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 03:36:05 (1787038565)
[ 5274.605537] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5276.278704] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 03:36:09 (1787038569)
[ 5277.817658] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5279.527203] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 03:36:12 (1787038572)
[ 5281.075133] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5283.055389] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 03:36:16 (1787038576)
[ 5284.674355] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5286.290180] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 03:36:19 (1787038579)
[ 5287.915968] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5290.026448] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 03:36:22 (1787038582)
[ 5291.769705] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5293.228930] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 03:36:26 (1787038586)
[ 5295.182838] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5297.408965] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 03:36:30 (1787038590)
[ 5299.553727] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5301.471989] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 03:36:34 (1787038594)
[ 5303.399825] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5305.617236] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 03:36:38 (1787038598)
[ 5308.675774] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5310.409821] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 03:36:43 (1787038603)
[ 5342.315340] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 03:37:15 (1787038635)
[ 5373.100574] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 03:37:46 (1787038666)
[ 5373.632393] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5384.518226] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 03:37:57 (1787038677)
[ 5384.910563] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5395.631270] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 03:38:08 (1787038688)
[ 5396.108490] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5405.648527] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 03:38:18 (1787038698)
[ 5406.161188] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5414.395786] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 03:38:27 (1787038707)
[ 5414.882737] Lustre: *** cfs_fail_loc=510, val=3***
[ 5424.104404] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 03:38:37 (1787038717)
[ 5424.422312] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5434.959127] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 03:38:48 (1787038728)
[ 5435.344699] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5493.472313] Lustre: lustre-OST0001-osc-ffff8c9788fdd000: Connection to lustre-OST0001 (at 192.168.201.150@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5493.493060] Lustre: Skipped 12 previous similar messages
[ 5501.224930] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 03:39:54 (1787038794)
[ 5501.710621] LustreError: 82713:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5504.908809] LustreError: lustre-MDT0000-mdc-ffff8c9788fdd000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5504.936352] LustreError: 82728:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8c9788fdd000: inode [0x20000a042:0xe:0x0] mdc close failed: rc = -108
[ 5504.971130] LustreError: 82728:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8c9788fdd000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8c978990ff00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5504.985253] LustreError: 82733:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5504.994942] Lustre: lustre-MDT0000-mdc-ffff8c9788fdd000: Connection restored to 192.168.201.150@tcp (at 192.168.201.150@tcp)
[ 5504.997563] LustreError: 82733:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5505.008686] Lustre: Skipped 11 previous similar messages
[ 5505.775130] LustreError: 82713:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5507.948234] LustreError: 82744:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5511.195152] LustreError: lustre-MDT0000-mdc-ffff8c9788fdd000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5511.213339] LustreError: 82758:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8c9788fdd000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8c97865a4100) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5511.244767] LustreError: 82758:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5511.254387] LustreError: 82763:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5512.039128] LustreError: 82744:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5519.107903] LustreError: lustre-MDT0000-mdc-ffff8c9788fdd000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5519.648983] LustreError: 82799:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5522.839992] LustreError: lustre-MDT0000-mdc-ffff8c9788fdd000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5522.925434] LustreError: 82819:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5523.719133] LustreError: 82799:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5523.728967] LustreError: 82799:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8c9788fdd000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5530.181501] LustreError: 82874:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5533.240223] LustreError: lustre-MDT0000-mdc-ffff8c9788fdd000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5533.301292] LustreError: 82888:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8c9788fdd000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8c97898c7300) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5533.336449] LustreError: 82888:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5533.350717] LustreError: 82893:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5533.363705] LustreError: 82893:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5534.208279] LustreError: 82874:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5541.531198] LustreError: lustre-MDT0000-mdc-ffff8c9788fdd000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5542.021141] LustreError: 82929:0:(ldlm_flock.c:858:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5544.905382] LustreError: lustre-MDT0000-mdc-ffff8c9788fdd000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5546.118569] LustreError: 82929:0:(ldlm_flock.c:858:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5549.232640] LustreError: lustre-MDT0000-mdc-ffff8c9788fdd000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5549.297190] LustreError: 82977:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5550.315593] LustreError: 82958:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8c9788fdd000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5558.454125] LustreError: 83029:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5558.468458] LustreError: 83029:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5562.085058] LustreError: lustre-MDT0000-mdc-ffff8c9788fdd000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5562.457177] LustreError: 83029:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5562.470021] LustreError: 83029:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5562.482120] LustreError: 83029:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8c9788fdd000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5565.660432] LustreError: 83080:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5565.678156] LustreError: 83080:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 46 previous similar messages
[ 5573.857321] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 03:41:06 (1787038866)
[ 5592.554457] LustreError: 79465:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5592.569647] LustreError: 79465:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5596.639144] LustreError: 79465:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5596.654561] LustreError: 79465:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5603.844789] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 03:41:36 (1787038896)
[ 5654.233597] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 03:42:27 (1787038947)
[ 5719.483259] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 03:43:32 (1787039012)
[ 5753.125921] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 03:44:06 (1787039046)
[ 5753.455574] Lustre: Mounted lustre-client
[ 5876.553093] Lustre: Unmounted lustre-client
[ 5883.144948] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 03:46:16 (1787039176)
[ 5883.637776] LustreError: 87357:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5885.942377] LustreError: lustre-OST0000-osc-ffff8c9788fdd000: operation ldlm_enqueue to node 192.168.201.150@tcp failed: rc = -107
[ 5885.949797] LustreError: Skipped 9 previous similar messages
[ 5885.967975] LustreError: lustre-OST0000-osc-ffff8c9788fdd000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5885.999851] LustreError: 87439:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff8c9788fdd000: namespace resource [0x240000400:0x40cd:0x0].0x0 (ffff8c978990fa00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5886.019404] LustreError: 87439:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 3 previous similar messages
[ 5886.359173] LustreError: 87357:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5886.363678] Lustre: 2359:0:(llite_lib.c:4327:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.150@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5891.628837] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 03:46:24 (1787039184)
[ 5949.407155] Lustre: 88023:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787039188/real 1787039188]  req@ffff8c9784768000 x1873840342217216/t0(0) o101->lustre-MDT0000-mdc-ffff8c9788fdd000@192.168.201.150@tcp:12/10 lens 328/344 e 0 to 1 dl 1787039243 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 5949.433055] Lustre: 88023:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 19 previous similar messages
[ 5955.168463] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 03:47:28 (1787039248)
[ 5956.274612] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 5957.747375] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 03:47:31 (1787039251)
[ 6019.437708] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6020.792187] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 03:48:34 (1787039314)
[ 6078.380676] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 03:49:32 (1787039372)
[ 6079.783810] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6080.826294] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 03:49:34 (1787039374)
[ 6081.874920] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6083.090623] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 03:49:36 (1787039376)
[ 6106.809738] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 03:50:00 (1787039400)
[ 6114.309513] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6133.730585] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 192.168.201.150@tcp) was lost; in progress operations using this service will fail
[ 6133.736286] LustreError: Skipped 2 previous similar messages
[ 6133.743849] Lustre: Evicted from MGS (at 192.168.201.150@tcp) after server handle changed from 0x437ef3b07a976161 to 0x437ef3b07a977a20
[ 6133.744676] Lustre: lustre-MDT0000-mdc-ffff8c9788fdd000: Connection to lustre-MDT0000 (at 192.168.201.150@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6133.756108] Lustre: Skipped 2 previous similar messages
[ 6133.767296] Lustre: MGC192.168.201.150@tcp: Connection restored to 192.168.201.150@tcp (at 192.168.201.150@tcp)
[ 6133.779950] Lustre: Skipped 16 previous similar messages
[ 6133.810610] Lustre: Skipped 14 previous similar messages
[ 6144.113727] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6145.191792] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6153.082046] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 03:50:46 (1787039446)
[ 6154.722972] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6155.714102] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 03:50:49 (1787039449)
[ 6172.007695] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 03:51:05 (1787039465)
[ 6190.052640] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 192.168.201.150@tcp) was lost; in progress operations using this service will fail
[ 6190.059070] LustreError: Skipped 1 previous similar message
[ 6190.070609] Lustre: Evicted from MGS (at 192.168.201.150@tcp) after server handle changed from 0x437ef3b07a977e41 to 0x437ef3b07a9781c8
[ 6190.076105] Lustre: Skipped 1 previous similar message
[ 6196.365241] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 03:51:30 (1787039490)
[ 6233.303223] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6235.419655] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6319.010211] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 192.168.201.150@tcp) was lost; in progress operations using this service will fail
[ 6328.536192] Lustre: Evicted from MGS (at 192.168.201.150@tcp) after server handle changed from 0x437ef3b07a9781c8 to 0x437ef3b07a978e9b
[ 6354.383815] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6356.511685] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6382.069462] Lustre: 68003:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.150@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6404.497772] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6406.080921] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6544.562950] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 03:57:17 (1787039837)
[ 6557.777653] LustreError: lustre-OST0000-osc-ffff8c9788fdd000: operation ost_setattr to node 192.168.201.150@tcp failed: rc = -107
[ 6557.789967] LustreError: Skipped 384 previous similar messages
[ 6622.509961] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6625.657181] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6892.058893] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 04:03:05 (1787040185)
[ 7045.613473] Lustre: lustre-MDT0000-mdc-ffff8c9788fdd000: Connection to lustre-MDT0000 (at 192.168.201.150@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7045.631733] Lustre: Skipped 6 previous similar messages
[ 7045.631896] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 192.168.201.150@tcp) was lost; in progress operations using this service will fail
[ 7045.652424] LustreError: Skipped 1 previous similar message
[ 7045.666980] Lustre: Evicted from MGS (at 192.168.201.150@tcp) after server handle changed from 0x437ef3b07a9791ce to 0x437ef3b07aa6e302
[ 7045.678988] Lustre: Skipped 1 previous similar message
[ 7045.694476] Lustre: MGC192.168.201.150@tcp: Connection restored to 192.168.201.150@tcp (at 192.168.201.150@tcp)
[ 7045.701808] Lustre: Skipped 9 previous similar messages
[ 7075.295168] Lustre: 2359:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787040329/real 1787040329]  req@ffff8c97ab210700 x1873840359558912/t0(0) o400->lustre-MDT0000-mdc-ffff8c9788fdd000@192.168.201.150@tcp:12/10 lens 224/224 e 0 to 1 dl 1787040369 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7075.321228] Lustre: 2359:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 22 previous similar messages
[ 7079.175711] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 04:06:12 (1787040372)
[ 7080.488618] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7082.140860] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 04:06:15 (1787040375)
[ 7084.048048] LustreError: lustre-MDT0000-mdc-ffff8c9788fdd000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7091.069088] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 04:06:24 (1787040384)
[ 7258.157926] LustreError: lustre-OST0000-osc-ffff8c9788fdd000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7260.472963] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 04:09:13 (1787040553)
[ 7303.211700] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 04:09:56 (1787040596)
[ 7304.488437] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7306.117377] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 04:09:59 (1787040599)
[ 7307.168439] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7308.710452] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 04:10:02 (1787040602)
[ 7340.409771] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 04:10:33 (1787040633)
[ 7378.915834] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 192.168.201.150@tcp) was lost; in progress operations using this service will fail
[ 7378.942510] Lustre: Evicted from MGS (at 192.168.201.150@tcp) after server handle changed from 0x437ef3b07aa6e302 to 0x437ef3b07aa70eb4
[ 7378.953477] Lustre: Skipped 1 previous similar message
[ 7389.486923] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 04:11:22 (1787040682)
[ 7390.806826] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7392.126597] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 04:11:25 (1787040685)
[ 7393.556230] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7395.010472] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 04:11:28 (1787040688)
[ 7407.279629] Lustre: Unmounted lustre-client
[ 7408.992930] Lustre: Mounted lustre-client
[ 7411.642958] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7438.972227] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2