[ 1805.440656] LustreError: lustre-OST0000-osc-ffff94c960aa0800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1805.459189] Lustre: 2364:0:(llite_lib.c:4327:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.113@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1812.645985] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 03:49:19 (1784188159)
[ 1814.519619] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1816.182582] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 03:49:23 (1784188163)
[ 1817.662373] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1819.125668] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 03:49:26 (1784188166)
[ 1822.249190] Lustre: lustre-MDT0000-mdc-ffff94c960aa0800: Connection to lustre-MDT0000 (at 192.168.203.113@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 1822.261054] Lustre: Skipped 14 previous similar messages
[ 1840.622122] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 192.168.203.113@tcp) was lost; in progress operations using this service will fail
[ 1840.645114] Lustre: Evicted from MGS (at 192.168.203.113@tcp) after server handle changed from 0x4ba8d4ca4bd89f8a to 0x4ba8d4ca4bd8b7b6
[ 1840.657182] Lustre: MGC192.168.203.113@tcp: Connection restored to 192.168.203.113@tcp (at 192.168.203.113@tcp)
[ 1840.665234] Lustre: Skipped 14 previous similar messages
[ 1842.500829] Lustre: 15917:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.113@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 1950.108580] LustreError: lustre-MDT0000-mdc-ffff94c960aa0800: operation mds_reint to node 192.168.203.113@tcp failed: rc = -19
[ 1950.117444] LustreError: Skipped 2 previous similar messages
[ 1968.613925] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 192.168.203.113@tcp) was lost; in progress operations using this service will fail
[ 1977.782589] Lustre: Evicted from MGS (at 192.168.203.113@tcp) after server handle changed from 0x4ba8d4ca4bd8b7b6 to 0x4ba8d4ca4bdadb76
[ 1992.516542] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 03:52:19 (1784188339)
[ 1993.057571] Lustre: *** cfs_fail_loc=305, val=0***
[ 1993.059856] Lustre: Skipped 4 previous similar messages
[ 2038.248125] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 192.168.203.113@tcp) was lost; in progress operations using this service will fail
[ 2038.275665] Lustre: Evicted from MGS (at 192.168.203.113@tcp) after server handle changed from 0x4ba8d4ca4bdadb76 to 0x4ba8d4ca4bdae0a8
[ 2050.097720] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2051.317091] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2058.850940] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 03:53:25 (1784188405)
[ 2074.092467] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 192.168.203.113@tcp) was lost; in progress operations using this service will fail
[ 2074.129694] Lustre: Evicted from MGS (at 192.168.203.113@tcp) after server handle changed from 0x4ba8d4ca4bdae0a8 to 0x4ba8d4ca4bdae5a2
[ 2074.175719] LustreError: lustre-MDT0000-mdc-ffff94c960aa0800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2092.770434] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 03:53:59 (1784188439)
[ 2104.813438] LustreError: lustre-OST0000-osc-ffff94c960aa0800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2121.565679] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 03:54:28 (1784188468)
[ 2134.806068] LustreError: lustre-MDT0000-mdc-ffff94c960aa0800: operation ldlm_enqueue to node 192.168.203.113@tcp failed: rc = -19
[ 2134.816722] LustreError: Skipped 1 previous similar message
[ 2150.879248] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 192.168.203.113@tcp) was lost; in progress operations using this service will fail
[ 2161.065207] Lustre: Evicted from MGS (at 192.168.203.113@tcp) after server handle changed from 0x4ba8d4ca4bdae5a2 to 0x4ba8d4ca4bdb3b70
[ 2177.230643] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2178.791902] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2262.530258] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 192.168.203.113@tcp) was lost; in progress operations using this service will fail
[ 2262.554457] Lustre: Evicted from MGS (at 192.168.203.113@tcp) after server handle changed from 0x4ba8d4ca4bdb3b70 to 0x4ba8d4ca4bdd2dab
[ 2263.878445] Lustre: 15917:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.113@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2288.316271] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2290.342355] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2355.273525] LustreError: lustre-MDT0000-mdc-ffff94c960aa0800: operation mds_reint to node 192.168.203.113@tcp failed: rc = -19
[ 2355.287224] LustreError: Skipped 1 previous similar message
[ 2374.129280] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 192.168.203.113@tcp) was lost; in progress operations using this service will fail
[ 2374.158910] Lustre: Evicted from MGS (at 192.168.203.113@tcp) after server handle changed from 0x4ba8d4ca4bdd2dab to 0x4ba8d4ca4bdf0797
[ 2375.542792] Lustre: 15917:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.113@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2401.425387] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2403.475849] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2432.700960] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 03:59:39 (1784188779)
[ 2436.794773] Lustre: lustre-MDT0000-mdc-ffff94c960aa0800: Connection to lustre-MDT0000 (at 192.168.203.113@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2436.808987] Lustre: Skipped 7 previous similar messages
[ 2456.044262] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 192.168.203.113@tcp) was lost; in progress operations using this service will fail
[ 2456.064205] Lustre: Evicted from MGS (at 192.168.203.113@tcp) after server handle changed from 0x4ba8d4ca4bdf0797 to 0x4ba8d4ca4bdfe193
[ 2456.088422] Lustre: MGC192.168.203.113@tcp: Connection restored to 192.168.203.113@tcp (at 192.168.203.113@tcp)
[ 2456.103536] Lustre: Skipped 14 previous similar messages
[ 2463.814495] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2495.818977] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2528.556918] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2567.989047] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2609.631523] Lustre: 2363:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784188941/real 1784188941]  req@ffff94c94689ad80 x1870855036081536/t0(0) o400->MGC192.168.203.113@tcp@192.168.203.113@tcp:26/25 lens 224/224 e 0 to 1 dl 1784188957 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2609.688778] Lustre: 2363:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
[ 2609.705308] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 192.168.203.113@tcp) was lost; in progress operations using this service will fail
[ 2609.736386] LustreError: Skipped 3 previous similar messages
[ 2609.766155] Lustre: Evicted from MGS (at 192.168.203.113@tcp) after server handle changed from 0x4ba8d4ca4be0f675 to 0x4ba8d4ca4be19338
[ 2609.780046] Lustre: Skipped 3 previous similar messages
[ 2617.403571] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2645.196128] LustreError: lustre-MDT0000-mdc-ffff94c960aa0800: operation mds_reint to node 192.168.203.113@tcp failed: rc = -19
[ 2645.206147] LustreError: Skipped 7 previous similar messages
[ 2671.227937] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2753.360980] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 04:05:00 (1784189100)
[ 2806.609768] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 2808.810653] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3098.678137] Lustre: lustre-OST0000-osc-ffff94c960aa0800: Connection to lustre-OST0000 (at 192.168.203.113@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3098.702246] Lustre: Skipped 7 previous similar messages
[ 3137.845106] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3140.067762] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3430.991045] LustreError: lustre-OST0000-osc-ffff94c960aa0800: operation ldlm_enqueue to node 192.168.203.113@tcp failed: rc = -19
[ 3430.995964] LustreError: Skipped 3 previous similar messages
[ 3469.304274] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3470.996341] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3727.002618] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 04:21:13 (1784190073)
[ 3743.711709] Lustre: 50424:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784190076/real 1784190076]  req@ffff94c947bf8380 x1870855056413696/t0(0) o101->lustre-MDT0000-mdc-ffff94c960aa0800@192.168.203.113@tcp:12/10 lens 576/1152 e 0 to 1 dl 1784190092 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3743.764135] Lustre: 50424:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 3743.788535] Lustre: lustre-MDT0000-mdc-ffff94c960aa0800: Connection to lustre-MDT0000 (at 192.168.203.113@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3743.806745] Lustre: Skipped 1 previous similar message
[ 3743.849979] Lustre: lustre-MDT0000-mdc-ffff94c960aa0800: Connection restored to 192.168.203.113@tcp (at 192.168.203.113@tcp)
[ 3743.856897] Lustre: Skipped 14 previous similar messages
[ 3753.333866] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 04:21:40 (1784190100)
[ 3779.275580] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 04:22:06 (1784190126)
[ 3803.787395] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 04:22:30 (1784190150)
[ 3804.150570] Lustre: Mounted lustre-client
[ 3834.852610] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 192.168.203.113@tcp) was lost; in progress operations using this service will fail
[ 3834.867098] LustreError: Skipped 2 previous similar messages
[ 3834.902521] Lustre: Evicted from MGS (at 192.168.203.113@tcp) after server handle changed from 0x4ba8d4ca4be32078 to 0x4ba8d4ca4bf94c75
[ 3834.918033] Lustre: Skipped 2 previous similar messages
[ 3834.927515] Lustre: MGC192.168.203.113@tcp: Connection restored to 192.168.203.113@tcp (at 192.168.203.113@tcp)
[ 3834.943757] Lustre: Skipped 2 previous similar messages
[ 3847.740035] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 3849.244198] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3851.415838] Lustre: Unmounted lustre-client
[ 3857.956569] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 04:23:24 (1784190204)
[ 3995.645287] Lustre: lustre-OST0000-osc-ffff94c960aa0800: Connection restored to 192.168.203.113@tcp (at 192.168.203.113@tcp)
[ 3995.656289] Lustre: Skipped 8 previous similar messages
[ 4154.182169] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 04:28:21 (1784190501)
[ 4202.227947] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 04:29:09 (1784190549)
[ 4204.859225] Lustre: Unmounted lustre-client
[ 4226.134238] Lustre: Mounted lustre-client
[ 4233.339020] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 04:29:40 (1784190580)
[ 4233.607970] LustreError: 56336:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4234.658172] LustreError: 56336:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4234.926491] Lustre: *** cfs_fail_loc=305, val=0***
[ 4257.487797] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 04:30:04 (1784190604)
[ 4258.145363] Lustre: Mounted lustre-client
[ 4260.080777] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4270.427161] Lustre: Unmounted lustre-client
[ 4277.621449] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 04:30:24 (1784190624)
[ 4365.733136] LustreError: lustre-MDT0000-mdc-ffff94c948267800: operation mds_reint to node 192.168.203.113@tcp failed: rc = -19
[ 4365.746389] Lustre: lustre-MDT0000-mdc-ffff94c948267800: Connection to lustre-MDT0000 (at 192.168.203.113@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4365.767954] Lustre: Skipped 23 previous similar messages
[ 4383.199205] Lustre: 2364:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784190715/real 1784190715]  req@ffff94c958229880 x1870855059125504/t0(0) o400->MGC192.168.203.113@tcp@192.168.203.113@tcp:26/25 lens 224/224 e 0 to 1 dl 1784190731 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4383.234042] Lustre: 2364:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 160 previous similar messages
[ 4383.242495] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 192.168.203.113@tcp) was lost; in progress operations using this service will fail
[ 4393.452923] Lustre: Evicted from MGS (at 192.168.203.113@tcp) after server handle changed from 0x4ba8d4ca4bf968d0 to 0x4ba8d4ca4bfc17e8
[ 4393.472985] Lustre: MGC192.168.203.113@tcp: Connection restored to 192.168.203.113@tcp (at 192.168.203.113@tcp)
[ 4393.484875] Lustre: Skipped 11 previous similar messages
[ 4553.887858] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 04:35:00 (1784190900)
[ 4558.377977] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4571.629494] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 192.168.203.113@tcp) was lost; in progress operations using this service will fail
[ 4571.661716] Lustre: Evicted from MGS (at 192.168.203.113@tcp) after server handle changed from 0x4ba8d4ca4bfc17e8 to 0x4ba8d4ca4c00edd7
[ 4571.731753] LustreError: lustre-MDT0000-mdc-ffff94c948267800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4585.151024] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 04:35:32 (1784190932)
[ 4585.542841] Lustre: Mounted lustre-client
[ 4592.678326] LustreError: lustre-OST0000-osc-ffff94c948267800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4592.704912] LustreError: 60036:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff94c948267800: namespace resource [0x240000400:0x4383:0x0].0x0 (ffff94c9509d4b00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4605.276642] Lustre: Unmounted lustre-client
[ 4612.405462] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 04:35:59 (1784190959)
[ 4618.706807] LustreError: lustre-MDT0000-mdc-ffff94c948267800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4618.735083] LustreError: 60708:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4625.744477] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 04:36:12 (1784190972)
[ 4625.943524] LustreError: 61411:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4631.008494] LustreError: 61411:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=0
[ 4631.021788] LustreError: 61411:0:(import.c:711:ptlrpc_connect_import_locked()) already connected
[ 4631.446041] LustreError: lustre-MDT0000-mdc-ffff94c948267800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4631.463294] LustreError: 61428:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4631.532890] LustreError: 61433:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4632.633797] LustreError: 61439:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4632.644210] LustreError: 61439:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4634.933678] LustreError: 61461:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4634.942361] LustreError: 61461:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4639.592089] LustreError: 61506:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4639.609340] LustreError: 61506:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4648.575243] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 04:36:35 (1784190995)
[ 4685.147822] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4686.958673] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4696.995857] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 04:37:23 (1784191043)
[ 4734.752852] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4736.337906] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4745.940062] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 04:38:13 (1784191093)
[ 4805.971405] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4807.451280] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4815.495084] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 04:39:22 (1784191162)
[ 4852.162775] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4853.501757] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4858.694249] Lustre: Unmounted lustre-client
[ 4909.308891] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4912.369045] Lustre: Mounted lustre-client
[ 4919.664294] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 04:41:06 (1784191266)
[ 4922.175482] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 4923.904643] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 04:41:10 (1784191270)
[ 4952.577402] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 04:41:39 (1784191299)
[ 4954.062196] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 4955.676789] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 04:41:42 (1784191302)
[ 4957.200265] Lustre: *** cfs_fail_loc=805, val=0***
[ 4957.246620] Lustre: Mounted lustre-client
[ 4962.135175] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4982.751509] Lustre: lustre-MDT0000-mdc-ffff94c950c6e000: Connection to lustre-MDT0000 (at 192.168.203.113@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4982.777616] Lustre: Skipped 13 previous similar messages
[ 4982.779528] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 192.168.203.113@tcp) was lost; in progress operations using this service will fail
[ 4982.836343] Lustre: Evicted from MGS (at 192.168.203.113@tcp) after server handle changed from 0x4ba8d4ca4c00f7d2 to 0x4ba8d4ca4c00fb6e
[ 4987.680841] Lustre: 2363:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784191320/real 1784191320]  req@ffff94c9503d6a00 x1870855061198080/t0(0) o400->lustre-MDT0000-mdc-ffff94c950c6e000@192.168.203.113@tcp:12/10 lens 224/224 e 0 to 1 dl 1784191336 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4987.730852] Lustre: 2363:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
[ 4989.836086] Lustre: Unmounted lustre-client
[ 4996.538831] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 04:42:23 (1784191343)
[ 5019.129606] Lustre: MGC192.168.203.113@tcp: Connection restored to 192.168.203.113@tcp (at 192.168.203.113@tcp)
[ 5019.137532] Lustre: Skipped 16 previous similar messages
[ 5021.059634] Lustre: 68027:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.113@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 5039.726224] LustreError: 2362:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff94c95035d500 x1870855061207424/t94489280516(94489280516) o101->lustre-MDT0000-mdc-ffff94c950c6e000@192.168.203.113@tcp:12/10 lens 664/608 e 0 to 0 dl 1784191403 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5044.590958] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5045.455948] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5053.054729] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 04:43:19 (1784191399)
[ 5054.485378] LustreError: lustre-OST0000-osc-ffff94c950c6e000: operation ost_write to node 192.168.203.113@tcp failed: rc = -107
[ 5054.495627] LustreError: Skipped 3 previous similar messages
[ 5054.531343] LustreError: lustre-OST0000-osc-ffff94c950c6e000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5054.544807] Lustre: 2365:0:(llite_lib.c:4327:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.113@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5054.570705] LustreError: 72551:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff94c950c6e000: namespace resource [0x240000400:0x43e2:0x0].0x0 (ffff94c9603de100) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5062.647142] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 04:43:29 (1784191409)
[ 5064.342595] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5066.252607] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 04:43:33 (1784191413)
[ 5068.189465] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5069.832417] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 04:43:36 (1784191416)
[ 5072.037959] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5074.254328] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 04:43:40 (1784191420)
[ 5076.090629] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5077.634548] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 04:43:44 (1784191424)
[ 5079.132988] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5081.023458] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 04:43:47 (1784191427)
[ 5082.673672] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5084.278193] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 04:43:51 (1784191431)
[ 5085.601563] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5087.571355] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 04:43:54 (1784191434)
[ 5089.116510] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5090.692211] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 04:43:57 (1784191437)
[ 5092.279372] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5093.834219] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 04:44:00 (1784191440)
[ 5095.182759] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5097.139675] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 04:44:04 (1784191444)
[ 5098.993757] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5100.583229] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 04:44:07 (1784191447)
[ 5102.942827] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5104.256916] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 04:44:11 (1784191451)
[ 5131.849406] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 04:44:38 (1784191478)
[ 5160.098622] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 04:45:07 (1784191507)
[ 5160.447757] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5169.829770] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 04:45:16 (1784191516)
[ 5170.129351] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5180.352275] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 04:45:27 (1784191527)
[ 5180.693339] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5188.383846] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 04:45:35 (1784191535)
[ 5188.652114] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5196.952613] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 04:45:43 (1784191543)
[ 5197.252733] Lustre: *** cfs_fail_loc=510, val=3***
[ 5204.834589] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 04:45:51 (1784191551)
[ 5205.254455] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5215.906727] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 04:46:02 (1784191562)
[ 5216.428289] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5280.887200] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 04:47:08 (1784191628)
[ 5281.194123] LustreError: 82736:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5284.038948] LustreError: lustre-MDT0000-mdc-ffff94c950c6e000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5284.080274] LustreError: 82750:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff94c950c6e000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff94c96031bb00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5285.263760] LustreError: 82736:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5287.390853] LustreError: 82766:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5290.294967] LustreError: lustre-MDT0000-mdc-ffff94c950c6e000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5290.320651] LustreError: 82780:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff94c950c6e000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff94c9509d4700) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5290.349748] LustreError: 82780:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5291.479133] LustreError: 82766:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5298.363893] LustreError: lustre-MDT0000-mdc-ffff94c950c6e000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5298.810754] LustreError: 82822:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5301.628686] LustreError: lustre-MDT0000-mdc-ffff94c950c6e000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5301.690794] LustreError: 82841:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5301.697566] LustreError: 82841:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5302.771843] LustreError: 82848:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5302.781684] LustreError: 82848:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5302.811037] LustreError: 82822:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5302.819097] LustreError: 82822:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff94c950c6e000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5304.997222] LustreError: 82870:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5305.024981] LustreError: 82870:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5308.682654] LustreError: 82897:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5311.662308] LustreError: lustre-MDT0000-mdc-ffff94c950c6e000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5311.680198] LustreError: 82911:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff94c950c6e000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff94c9509d4b00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5311.709271] LustreError: 82911:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5312.759681] LustreError: 82897:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5319.831265] LustreError: lustre-MDT0000-mdc-ffff94c950c6e000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5320.176711] LustreError: 82953:0:(ldlm_flock.c:858:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5322.904331] LustreError: lustre-MDT0000-mdc-ffff94c950c6e000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5322.915401] LustreError: 82967:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff94c950c6e000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff94c9509d4e00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5322.922942] LustreError: 82967:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5324.239124] LustreError: 82953:0:(ldlm_flock.c:858:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5327.513677] LustreError: lustre-MDT0000-mdc-ffff94c950c6e000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5327.624413] LustreError: 83000:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5327.639173] LustreError: 83000:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5328.455828] LustreError: 82981:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff94c950c6e000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5336.594795] LustreError: 83052:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5336.609159] LustreError: 83052:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5339.884629] LustreError: lustre-MDT0000-mdc-ffff94c950c6e000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5339.937165] LustreError: 83074:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5339.945238] LustreError: 83074:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5340.687135] LustreError: 83052:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5340.693531] LustreError: 83052:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5340.701732] LustreError: 83052:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff94c950c6e000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5350.909807] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 04:48:17 (1784191697)
[ 5369.178210] LustreError: 70526:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5369.182510] LustreError: 70526:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5373.247173] LustreError: 70526:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5373.258595] LustreError: 70526:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5378.769874] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 04:48:46 (1784191726)
[ 5427.760957] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 04:49:34 (1784191774)
[ 5490.843615] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 04:50:38 (1784191838)
[ 5523.534293] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 04:51:10 (1784191870)
[ 5524.002759] Lustre: Mounted lustre-client
[ 5647.140866] Lustre: Unmounted lustre-client
[ 5655.174980] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 04:53:21 (1784192001)
[ 5655.983958] LustreError: 87379:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5658.678139] LustreError: lustre-OST0000-osc-ffff94c950c6e000: operation ldlm_enqueue to node 192.168.203.113@tcp failed: rc = -107
[ 5658.688852] LustreError: Skipped 9 previous similar messages
[ 5658.692102] Lustre: lustre-OST0000-osc-ffff94c950c6e000: Connection to lustre-OST0000 (at 192.168.203.113@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5658.708595] Lustre: Skipped 17 previous similar messages
[ 5658.720568] LustreError: lustre-OST0000-osc-ffff94c950c6e000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5658.738535] LustreError: 87461:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff94c950c6e000: namespace resource [0x240000400:0x440d:0x0].0x0 (ffff94c9509d4100) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5658.776509] LustreError: 87461:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5659.023114] LustreError: 87379:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5659.040392] Lustre: 2363:0:(llite_lib.c:4327:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.113@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5663.755721] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 04:53:31 (1784192011)
[ 5721.055283] Lustre: 88045:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784192014/real 1784192014]  req@ffff94c95035ea00 x1870855061378688/t0(0) o101->lustre-MDT0000-mdc-ffff94c950c6e000@192.168.203.113@tcp:12/10 lens 328/344 e 0 to 1 dl 1784192069 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 5721.097728] Lustre: 88045:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 13 previous similar messages
[ 5721.140530] Lustre: lustre-MDT0000-mdc-ffff94c950c6e000: Connection restored to 192.168.203.113@tcp (at 192.168.203.113@tcp)
[ 5721.148290] Lustre: Skipped 15 previous similar messages
[ 5727.737743] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 04:54:34 (1784192074)
[ 5729.105905] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 5731.030220] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 04:54:37 (1784192077)
[ 5794.469908] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 5795.697793] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 04:55:43 (1784192143)
[ 5853.726953] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 04:56:40 (1784192200)
[ 5855.518187] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 5857.023653] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 04:56:44 (1784192204)
[ 5858.466921] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 5860.287543] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 04:56:47 (1784192207)
[ 5894.840846] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 04:57:21 (1784192241)
[ 5904.723035] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5928.927237] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 192.168.203.113@tcp) was lost; in progress operations using this service will fail
[ 5928.935531] LustreError: Skipped 2 previous similar messages
[ 5928.957464] Lustre: Evicted from MGS (at 192.168.203.113@tcp) after server handle changed from 0x4ba8d4ca4c01062d to 0x4ba8d4ca4c011f39
[ 5928.977208] Lustre: Skipped 2 previous similar messages
[ 5941.045762] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5942.616902] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5952.279950] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 04:58:19 (1784192299)
[ 5954.575985] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 5956.051812] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 04:58:23 (1784192303)
[ 5969.896249] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 192.168.203.113@tcp) was lost; in progress operations using this service will fail
[ 5969.938996] Lustre: Evicted from MGS (at 192.168.203.113@tcp) after server handle changed from 0x4ba8d4ca4c011f39 to 0x4ba8d4ca4c0122f1
[ 5978.600633] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 04:58:45 (1784192325)
[ 6001.841638] Lustre: 68027:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.113@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6024.532316] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 04:59:31 (1784192371)
[ 6075.696715] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6078.775208] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6164.447316] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 192.168.203.113@tcp) was lost; in progress operations using this service will fail
[ 6164.453665] LustreError: Skipped 1 previous similar message
[ 6173.753403] Lustre: Evicted from MGS (at 192.168.203.113@tcp) after server handle changed from 0x4ba8d4ca4c0124fe to 0x4ba8d4ca4c013638
[ 6173.772522] Lustre: Skipped 1 previous similar message
[ 6179.361578] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6180.878780] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6213.892194] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6215.746746] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6328.522913] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 05:04:35 (1784192675)
[ 6340.038849] LustreError: lustre-OST0000-osc-ffff94c950c6e000: operation ost_setattr to node 192.168.203.113@tcp failed: rc = -107
[ 6340.042057] Lustre: lustre-OST0000-osc-ffff94c950c6e000: Connection to lustre-OST0000 (at 192.168.203.113@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6340.054897] LustreError: Skipped 39 previous similar messages
[ 6340.081830] Lustre: Skipped 9 previous similar messages
[ 6381.618217] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6384.117435] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6621.485426] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 05:09:28 (1784192968)
[ 6750.181874] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 192.168.203.113@tcp) was lost; in progress operations using this service will fail
[ 6750.197121] LustreError: Skipped 1 previous similar message
[ 6750.224109] Lustre: Evicted from MGS (at 192.168.203.113@tcp) after server handle changed from 0x4ba8d4ca4c0137ff to 0x4ba8d4ca4c10d92e
[ 6750.233599] Lustre: Skipped 1 previous similar message
[ 6750.243542] Lustre: MGC192.168.203.113@tcp: Connection restored to 192.168.203.113@tcp (at 192.168.203.113@tcp)
[ 6750.249805] Lustre: Skipped 12 previous similar messages
[ 6781.316222] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 05:12:08 (1784193128)
[ 6782.654829] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 6784.083478] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 05:12:11 (1784193131)
[ 6785.841423] LustreError: lustre-MDT0000-mdc-ffff94c950c6e000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6791.164200] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 05:12:18 (1784193138)
[ 6957.101835] LustreError: lustre-OST0000-osc-ffff94c950c6e000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6958.438125] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 05:15:05 (1784193305)
[ 6983.647226] Lustre: 2366:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784193311/real 1784193311]  req@ffff94c949495500 x1870855079080320/t0(0) o4->lustre-OST0000-osc-ffff94c950c6e000@192.168.203.113@tcp:6/4 lens 4584/448 e 0 to 1 dl 1784193331 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 6983.661225] Lustre: 2366:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 21 previous similar messages
[ 6983.666476] Lustre: lustre-OST0000-osc-ffff94c950c6e000: Connection to lustre-OST0000 (at 192.168.203.113@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6983.681448] Lustre: Skipped 3 previous similar messages
[ 6999.690222] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 05:15:46 (1784193346)
[ 7000.879762] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7002.830916] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 05:15:49 (1784193349)
[ 7004.660719] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7007.014954] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 05:15:53 (1784193353)
[ 7041.647461] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 05:16:28 (1784193388)
[ 7079.916746] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 192.168.203.113@tcp) was lost; in progress operations using this service will fail
[ 7079.928748] Lustre: Evicted from MGS (at 192.168.203.113@tcp) after server handle changed from 0x4ba8d4ca4c10d92e to 0x4ba8d4ca4c1101e5
[ 7079.934371] Lustre: Skipped 1 previous similar message
[ 7090.800418] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 05:17:17 (1784193437)
[ 7092.211724] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7093.673858] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 05:17:20 (1784193440)
[ 7095.257931] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7096.606799] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 05:17:23 (1784193443)
[ 7109.705201] Lustre: Unmounted lustre-client
[ 7111.294246] Lustre: Mounted lustre-client
[ 7113.978411] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7133.152050] Lustre: 2364:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784193465/real 1784193465]  req@ffff94c946cbea00 x1870855079266944/t0(0) o400->MGC192.168.203.113@tcp@192.168.203.113@tcp:26/25 lens 224/224 e 0 to 1 dl 1784193481 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7151.893429] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2