[ 1649.636205] LustreError: lustre-OST0000-osc-ffff89548b426800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1649.643106] Lustre: 2370:0:(llite_lib.c:4187:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.122@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1652.472413] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 10:51:17 (1780325477)
[ 1653.481328] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1654.181506] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 10:51:19 (1780325479)
[ 1654.948570] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1655.805045] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 10:51:21 (1780325481)
[ 1673.194521] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 192.168.202.122@tcp) was lost; in progress operations using this service will fail
[ 1673.208087] Lustre: Evicted from MGS (at 192.168.202.122@tcp) after server handle changed from 0x555c39731cbc46bc to 0x555c39731cbc6635
[ 1767.257846] LustreError: lustre-MDT0000-mdc-ffff89548b426800: operation ldlm_enqueue to node 192.168.202.122@tcp failed: rc = -19
[ 1767.261441] LustreError: Skipped 2 previous similar messages
[ 1784.802784] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 192.168.202.122@tcp) was lost; in progress operations using this service will fail
[ 1784.812302] Lustre: Evicted from MGS (at 192.168.202.122@tcp) after server handle changed from 0x555c39731cbc6635 to 0x555c39731cc07315
[ 1787.681982] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 10:53:33 (1780325613)
[ 1787.856893] Lustre: *** cfs_fail_loc=305, val=0***
[ 1787.858697] Lustre: Skipped 2 previous similar messages
[ 1820.642938] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 192.168.202.122@tcp) was lost; in progress operations using this service will fail
[ 1820.650388] Lustre: Evicted from MGS (at 192.168.202.122@tcp) after server handle changed from 0x555c39731cc07315 to 0x555c39731cc07871
[ 1823.439369] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1824.057983] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1827.872548] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 10:54:13 (1780325653)
[ 1836.015467] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 192.168.202.122@tcp) was lost; in progress operations using this service will fail
[ 1836.022929] Lustre: Evicted from MGS (at 192.168.202.122@tcp) after server handle changed from 0x555c39731cc07871 to 0x555c39731cc07de2
[ 1836.032420] LustreError: lustre-MDT0000-mdc-ffff89548b426800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 1843.515273] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 10:54:28 (1780325668)
[ 1849.837240] LustreError: lustre-OST0000-osc-ffff89548b426800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1857.966987] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 10:54:43 (1780325683)
[ 1869.639202] LustreError: lustre-MDT0000-mdc-ffff89548b426800: operation mds_reint to node 192.168.202.122@tcp failed: rc = -19
[ 1886.181454] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 192.168.202.122@tcp) was lost; in progress operations using this service will fail
[ 1886.197289] Lustre: Evicted from MGS (at 192.168.202.122@tcp) after server handle changed from 0x555c39731cc07de2 to 0x555c39731cc10fbc
[ 1888.618808] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1889.430370] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1968.098941] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 192.168.202.122@tcp) was lost; in progress operations using this service will fail
[ 1968.110456] Lustre: Evicted from MGS (at 192.168.202.122@tcp) after server handle changed from 0x555c39731cc10fbc to 0x555c39731cc43269
[ 1974.113174] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1975.272246] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2038.229660] LustreError: lustre-MDT0000-mdc-ffff89548b426800: operation ldlm_enqueue to node 192.168.202.122@tcp failed: rc = -19
[ 2038.234693] LustreError: Skipped 3 previous similar messages
[ 2054.117090] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 192.168.202.122@tcp) was lost; in progress operations using this service will fail
[ 2054.126972] Lustre: Evicted from MGS (at 192.168.202.122@tcp) after server handle changed from 0x555c39731cc43269 to 0x555c39731cc7299c
[ 2060.208143] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2061.161356] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2086.337856] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 10:58:31 (1780325911)
[ 2107.550131] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2124.774977] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 192.168.202.122@tcp) was lost; in progress operations using this service will fail
[ 2124.784951] LustreError: Skipped 1 previous similar message
[ 2124.797360] Lustre: Evicted from MGS (at 192.168.202.122@tcp) after server handle changed from 0x555c39731cc853e7 to 0x555c39731cc8595f
[ 2124.802666] Lustre: Skipped 1 previous similar message
[ 2124.804852] Lustre: MGC192.168.202.122@tcp: Connection restored to 192.168.202.122@tcp (at 192.168.202.122@tcp)
[ 2124.807835] Lustre: Skipped 24 previous similar messages
[ 2127.534777] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2133.795811] Lustre: lustre-MDT0000-mdc-ffff89548b426800: Connection to lustre-MDT0000 (at 192.168.202.122@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2133.802835] Lustre: Skipped 17 previous similar messages
[ 2152.581352] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2181.768622] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2221.446040] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2265.061506] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 192.168.202.122@tcp) was lost; in progress operations using this service will fail
[ 2265.074098] LustreError: Skipped 3 previous similar messages
[ 2265.094169] Lustre: Evicted from MGS (at 192.168.202.122@tcp) after server handle changed from 0x555c39731cca69c3 to 0x555c39731ccb99be
[ 2265.101177] Lustre: Skipped 3 previous similar messages
[ 2265.619195] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2297.865165] LustreError: lustre-MDT0000-mdc-ffff89548b426800: operation ldlm_enqueue to node 192.168.202.122@tcp failed: rc = -19
[ 2297.872180] LustreError: Skipped 8 previous similar messages
[ 2338.880343] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 11:02:44 (1780326164)
[ 2377.149351] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2378.453322] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 2704.411127] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2705.311023] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3017.206714] LustreError: lustre-OST0000-osc-ffff89548b426800: operation ldlm_enqueue to node 192.168.202.122@tcp failed: rc = -107
[ 3017.219389] LustreError: Skipped 2 previous similar messages
[ 3017.231075] Lustre: lustre-OST0000-osc-ffff89548b426800: Connection to lustre-OST0000 (at 192.168.202.122@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3017.258306] Lustre: Skipped 6 previous similar messages
[ 3039.279725] Lustre: lustre-OST0000-osc-ffff89548b426800: Connection restored to 192.168.202.122@tcp (at 192.168.202.122@tcp)
[ 3039.295493] Lustre: Skipped 13 previous similar messages
[ 3051.814940] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3054.470861] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3309.552900] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 11:18:54 (1780327134)
[ 3326.943577] Lustre: 50245:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780327136/real 1780327136]  req@ffff895483429c00 x1866804966017920/t0(0) o101->lustre-MDT0000-mdc-ffff89548b426800@192.168.202.122@tcp:12/10 lens 576/1152 e 0 to 1 dl 1780327152 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3326.990167] Lustre: 50245:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
[ 3334.240602] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 11:19:19 (1780327159)
[ 3358.724452] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 11:19:43 (1780327183)
[ 3381.891377] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 11:20:06 (1780327206)
[ 3382.325810] Lustre: Mounted lustre-client
[ 3412.975157] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 192.168.202.122@tcp) was lost; in progress operations using this service will fail
[ 3412.993646] LustreError: Skipped 1 previous similar message
[ 3413.029677] Lustre: Evicted from MGS (at 192.168.202.122@tcp) after server handle changed from 0x555c39731cccfdf3 to 0x555c39731cf19fa7
[ 3413.039906] Lustre: Skipped 1 previous similar message
[ 3413.983223] Lustre: 2370:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780327223/real 1780327223]  req@ffff8954812e9180 x1866804966037120/t0(0) o400->lustre-MDT0000-mdc-ffff895486106800@192.168.202.122@tcp:12/10 lens 224/224 e 0 to 1 dl 1780327239 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3414.013642] Lustre: 2370:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 3423.109562] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3424.272904] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3426.144348] LustreError: 52991:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff895486106800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 3426.167436] LustreError: 52991:0:(lov_obd.c:792:lov_cleanup()) Skipped 1 previous similar message
[ 3426.204154] Lustre: Unmounted lustre-client
[ 3431.307684] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 11:20:56 (1780327256)
[ 3566.560077] Lustre: 2370:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780327376/real 1780327376]  req@ffff8954812e8e00 x1866804966068864/t0(0) o4->lustre-OST0000-osc-ffff89548b426800@192.168.202.122@tcp:6/4 lens 488/448 e 0 to 1 dl 1780327392 ref 2 fl Rpc:XQr/602/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 3566.595299] Lustre: 2370:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 68 previous similar messages
[ 3630.048415] Lustre: lustre-OST0000-osc-ffff89548b426800: Connection to lustre-OST0000 (at 192.168.202.122@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3630.064128] Lustre: Skipped 16 previous similar messages
[ 3646.439921] Lustre: lustre-OST0000-osc-ffff89548b426800: Connection restored to 192.168.202.122@tcp (at 192.168.202.122@tcp)
[ 3646.449315] Lustre: Skipped 18 previous similar messages
[ 3720.009612] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 11:25:44 (1780327544)
[ 3766.870868] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 11:26:31 (1780327591)
[ 3769.687925] LustreError: 55141:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff89548b426800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 3769.714132] LustreError: 55141:0:(lov_obd.c:792:lov_cleanup()) Skipped 1 previous similar message
[ 3769.769230] Lustre: Unmounted lustre-client
[ 3783.905840] Lustre: Mounted lustre-client
[ 3788.732969] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 11:26:53 (1780327613)
[ 3788.890855] LustreError: 56121:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 3789.941455] LustreError: 56121:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 3790.027644] Lustre: *** cfs_fail_loc=305, val=0***
[ 3810.741874] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 11:27:15 (1780327635)
[ 3810.956892] Lustre: Mounted lustre-client
[ 3811.963304] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 3822.277693] Lustre: Unmounted lustre-client
[ 3826.209927] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 11:27:31 (1780327651)
[ 3881.901900] LustreError: lustre-MDT0000-mdc-ffff89548de91000: operation mds_reint to node 192.168.202.122@tcp failed: rc = -19
[ 3897.328802] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 192.168.202.122@tcp) was lost; in progress operations using this service will fail
[ 3897.354676] Lustre: Evicted from MGS (at 192.168.202.122@tcp) after server handle changed from 0x555c39731cf1b69f to 0x555c39731cf47e06
[ 3984.735680] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 11:30:09 (1780327809)
[ 3987.084978] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 3994.596701] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 192.168.202.122@tcp) was lost; in progress operations using this service will fail
[ 3994.611535] Lustre: Evicted from MGS (at 192.168.202.122@tcp) after server handle changed from 0x555c39731cf47e06 to 0x555c39731cf927da
[ 3994.657893] LustreError: lustre-MDT0000-mdc-ffff89548de91000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4006.518692] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 11:30:30 (1780327830)
[ 4007.630400] Lustre: Mounted lustre-client
[ 4016.286199] LustreError: lustre-OST0000-osc-ffff89548de91000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4026.335149] Lustre: 2371:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780327836/real 1780327836]  req@ffff895487a32a00 x1866804970701184/t0(0) o103->lustre-OST0000-osc-ffff89548ad85000@192.168.202.122@tcp:17/18 lens 328/224 e 0 to 1 dl 1780327852 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295
[ 4026.357161] Lustre: 2371:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 77 previous similar messages
[ 4030.818062] LustreError: 59836:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff89548ad85000: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 4030.838062] LustreError: 59836:0:(lov_obd.c:792:lov_cleanup()) Skipped 1 previous similar message
[ 4030.902937] Lustre: Unmounted lustre-client
[ 4041.197188] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 11:31:04 (1780327864)
[ 4048.603332] LustreError: lustre-MDT0000-mdc-ffff89548de91000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4048.648786] LustreError: 60478:0:(file.c:6098:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4060.555091] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 11:31:24 (1780327884)
[ 4060.938090] LustreError: 61184:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4065.419698] LustreError: lustre-MDT0000-mdc-ffff89548de91000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4065.458571] LustreError: 61200:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4065.481846] LustreError: 61184:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=456
[ 4065.498740] LustreError: 61184:0:(import.c:719:ptlrpc_connect_import_locked()) already connecting
[ 4065.546823] LustreError: 61205:0:(file.c:6098:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4066.685492] LustreError: 61212:0:(file.c:6098:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4066.701552] LustreError: 61212:0:(file.c:6098:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4069.190226] LustreError: 61234:0:(file.c:6098:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4069.202510] LustreError: 61234:0:(file.c:6098:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4074.015807] LustreError: 61279:0:(file.c:6098:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4074.030341] LustreError: 61279:0:(file.c:6098:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4086.771872] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 11:31:50 (1780327910)
[ 4127.170027] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4128.663689] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4139.133829] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 11:32:43 (1780327963)
[ 4174.678135] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4175.967923] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4186.314383] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 11:33:31 (1780328011)
[ 4250.216635] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4251.757710] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4259.767621] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 11:34:44 (1780328084)
[ 4264.445888] Lustre: lustre-OST0000-osc-ffff89548de91000: Connection to lustre-OST0000 (at 192.168.202.122@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4264.459118] Lustre: Skipped 19 previous similar messages
[ 4283.647775] Lustre: lustre-OST0000-osc-ffff89548de91000: Connection restored to 192.168.202.122@tcp (at 192.168.202.122@tcp)
[ 4283.655767] Lustre: Skipped 18 previous similar messages
[ 4293.711086] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4295.041558] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4300.783401] LustreError: 66850:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff89548de91000: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 4300.794932] LustreError: 66850:0:(lov_obd.c:792:lov_cleanup()) Skipped 1 previous similar message
[ 4300.842707] Lustre: Unmounted lustre-client
[ 4346.038446] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4349.183342] Lustre: Mounted lustre-client
[ 4355.890286] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 11:36:20 (1780328180)
[ 4358.882724] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 4361.073942] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 11:36:25 (1780328185)
[ 4400.821938] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 11:37:03 (1780328223)
[ 4403.230859] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 4406.544028] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 11:37:09 (1780328229)
[ 4408.674601] Lustre: *** cfs_fail_loc=805, val=0***
[ 4408.748538] Lustre: Mounted lustre-client
[ 4414.490547] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4435.423288] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 192.168.202.122@tcp) was lost; in progress operations using this service will fail
[ 4445.673133] Lustre: Evicted from MGS (at 192.168.202.122@tcp) after server handle changed from 0x555c39731cf931ce to 0x555c39731cf933f7
[ 4461.041692] LustreError: 2367:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff89548355f000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff895487ba5000) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4464.572895] LustreError: 70186:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff89548355f000: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 4464.580277] LustreError: 70186:0:(lov_obd.c:792:lov_cleanup()) Skipped 1 previous similar message
[ 4464.629115] Lustre: Unmounted lustre-client
[ 4472.794729] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 11:38:17 (1780328297)
[ 4499.306827] Lustre: 67685:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.202.122@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 4515.404168] LustreError: 2367:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff895487a31f80 x1866804970788480/t94489280516(94489280516) o101->lustre-MDT0000-mdc-ffff895485170000@192.168.202.122@tcp:12/10 lens 664/608 e 0 to 0 dl 1780328357 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 4518.066056] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4519.717508] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4528.412643] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 11:39:12 (1780328352)
[ 4529.709849] LustreError: lustre-OST0000-osc-ffff895485170000: operation ost_write to node 192.168.202.122@tcp failed: rc = -107
[ 4529.726265] LustreError: Skipped 3 previous similar messages
[ 4529.743430] LustreError: lustre-OST0000-osc-ffff895485170000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4529.764192] Lustre: 2371:0:(llite_lib.c:4187:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.122@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 4538.374374] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 11:39:22 (1780328362)
[ 4539.917842] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 4541.737432] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 11:39:26 (1780328366)
[ 4543.452106] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 4545.044343] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 11:39:29 (1780328369)
[ 4546.546227] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 4548.729961] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 11:39:33 (1780328373)
[ 4550.529895] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 4552.178235] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 11:39:36 (1780328376)
[ 4553.534210] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 4555.021373] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 11:39:39 (1780328379)
[ 4556.684538] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 4558.346351] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 11:39:43 (1780328383)
[ 4559.642364] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 4561.361918] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 11:39:46 (1780328386)
[ 4563.126919] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 4564.768670] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 11:39:49 (1780328389)
[ 4567.015509] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 4568.932120] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 11:39:53 (1780328393)
[ 4570.415980] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 4572.101951] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 11:39:56 (1780328396)
[ 4573.717767] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 4575.626772] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 11:40:00 (1780328400)
[ 4578.056621] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 4579.720885] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 11:40:04 (1780328404)
[ 4611.376793] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 11:40:36 (1780328436)
[ 4640.940811] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 11:41:05 (1780328465)
[ 4641.393539] Lustre: *** cfs_fail_loc=51b, val=3***
[ 4652.402837] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 11:41:17 (1780328477)
[ 4652.716564] Lustre: *** cfs_fail_loc=51b, val=4***
[ 4664.458986] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 11:41:28 (1780328488)
[ 4664.948907] Lustre: *** cfs_fail_loc=50f, val=3***
[ 4674.960991] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 11:41:39 (1780328499)
[ 4675.373220] Lustre: *** cfs_fail_loc=50f, val=4***
[ 4685.382684] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 11:41:49 (1780328509)
[ 4685.757406] Lustre: *** cfs_fail_loc=510, val=3***
[ 4697.250390] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 11:42:01 (1780328521)
[ 4697.784417] Lustre: *** cfs_fail_loc=51b, val=3***
[ 4705.696802] Lustre: 2369:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780328515/real 1780328515]  req@ffff895484a73b80 x1866804970829568/t0(0) o400->lustre-OST0001-osc-ffff895485170000@192.168.202.122@tcp:28/4 lens 224/224 e 0 to 1 dl 1780328531 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4705.732549] Lustre: 2369:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 18 previous similar messages
[ 4708.737923] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 11:42:13 (1780328533)
[ 4709.129952] Lustre: *** cfs_fail_loc=51c, val=3***
[ 4776.577365] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 11:43:20 (1780328600)
[ 4777.105357] LustreError: 82376:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 4780.160732] LustreError: lustre-MDT0000-mdc-ffff895485170000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4780.211356] LustreError: 82390:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff895485170000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff895487bc3e00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4780.250591] LustreError: 82390:0:(ldlm_resource.c:1180:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4781.175158] LustreError: 82376:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 4783.274414] LustreError: 82406:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 4786.060812] LustreError: lustre-MDT0000-mdc-ffff895485170000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4786.074280] LustreError: 82421:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff895485170000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff895488223000) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4786.092061] LustreError: 82421:0:(ldlm_resource.c:1180:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4787.343149] LustreError: 82406:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 4794.281214] LustreError: lustre-MDT0000-mdc-ffff895485170000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4794.789942] LustreError: 82462:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 4797.977622] LustreError: lustre-MDT0000-mdc-ffff895485170000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4798.065692] LustreError: 82482:0:(file.c:6098:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4798.073113] LustreError: 82482:0:(file.c:6098:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4798.887143] LustreError: 82462:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 4798.892594] LustreError: 82462:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff895485170000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 4799.207705] LustreError: 82488:0:(file.c:6098:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4799.227473] LustreError: 82488:0:(file.c:6098:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4801.613539] LustreError: 82511:0:(file.c:6098:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4801.631363] LustreError: 82511:0:(file.c:6098:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4805.243167] LustreError: 82537:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 4808.114744] LustreError: lustre-MDT0000-mdc-ffff895485170000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4808.145564] LustreError: 82552:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff895485170000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff895487bc3f00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4808.188833] LustreError: 82552:0:(ldlm_resource.c:1180:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4808.210655] LustreError: 82557:0:(file.c:6098:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4808.216788] LustreError: 82557:0:(file.c:6098:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 4809.335267] LustreError: 82537:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 4816.278466] LustreError: lustre-MDT0000-mdc-ffff895485170000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4816.722182] LustreError: 82594:0:(ldlm_flock.c:858:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 4819.722820] LustreError: lustre-MDT0000-mdc-ffff895485170000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4819.746707] LustreError: 82608:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff895485170000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff895488223800) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4819.772234] LustreError: 82608:0:(ldlm_resource.c:1180:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4820.799181] LustreError: 82594:0:(ldlm_flock.c:858:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 4823.990786] LustreError: lustre-MDT0000-mdc-ffff895485170000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4824.069711] LustreError: 82641:0:(file.c:6098:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4824.943785] LustreError: 82621:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff895485170000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 4833.077869] LustreError: 82693:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 4833.092855] LustreError: 82693:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 4836.742674] LustreError: lustre-MDT0000-mdc-ffff895485170000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4837.168786] LustreError: 82693:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 4837.179732] LustreError: 82693:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 4837.193294] LustreError: 82693:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff895485170000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 4840.318713] LustreError: 82744:0:(file.c:6098:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4840.327824] LustreError: 82744:0:(file.c:6098:ll_inode_revalidate_fini()) Skipped 46 previous similar messages
[ 4848.051979] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 11:44:32 (1780328672)
[ 4866.015251] Lustre: lustre-MDT0000-mdc-ffff895485170000: Connection to lustre-MDT0000 (at 192.168.202.122@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4866.032503] Lustre: Skipped 16 previous similar messages
[ 4867.401332] LustreError: 79127:0:(ldlm_lockd.c:2883:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 4867.409705] LustreError: 79127:0:(ldlm_lockd.c:2883:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 4871.479139] LustreError: 79127:0:(ldlm_lockd.c:2883:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 4871.486169] LustreError: 79127:0:(ldlm_lockd.c:2883:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 4876.954220] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 11:45:01 (1780328701)
[ 4919.784881] Lustre: lustre-MDT0000-mdc-ffff895485170000: Connection restored to 192.168.202.122@tcp (at 192.168.202.122@tcp)
[ 4919.799533] Lustre: Skipped 18 previous similar messages
[ 4927.498502] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 11:45:51 (1780328751)
[ 4992.015389] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 11:46:56 (1780328816)
[ 5024.713652] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 11:47:29 (1780328849)
[ 5025.343121] Lustre: Mounted lustre-client
[ 5148.119882] LustreError: 86405:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff89548dfe7000: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 5148.124846] LustreError: 86405:0:(lov_obd.c:792:lov_cleanup()) Skipped 1 previous similar message
[ 5148.163817] Lustre: Unmounted lustre-client
[ 5154.095156] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 11:49:38 (1780328978)
[ 5154.740225] LustreError: 87021:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5157.072194] LustreError: lustre-OST0000-osc-ffff895485170000: operation ldlm_enqueue to node 192.168.202.122@tcp failed: rc = -107
[ 5157.079635] LustreError: Skipped 9 previous similar messages
[ 5157.101188] LustreError: lustre-OST0000-osc-ffff895485170000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5157.112940] LustreError: 87104:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-OST0000-osc-ffff895485170000: namespace resource [0x240000400:0x65cd:0x0].0x0 (ffff8954858de200) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5157.130133] LustreError: 87104:0:(ldlm_resource.c:1180:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5157.471201] LustreError: 87021:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5157.475276] Lustre: 2371:0:(llite_lib.c:4187:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.122@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5163.579585] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 11:49:48 (1780328988)
[ 5224.696493] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 11:50:49 (1780329049)
[ 5225.843320] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 5227.045920] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 11:50:52 (1780329052)
[ 5288.433723] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 5289.612885] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 11:51:54 (1780329114)
[ 5345.248940] Lustre: 89417:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780329116/real 1780329116]  req@ffff895483673b80 x1866804970981120/t0(0) o36->lustre-MDT0000-mdc-ffff895485170000@192.168.202.122@tcp:12/10 lens 496/440 e 0 to 1 dl 1780329171 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ln.0' uid:0 gid:0 projid:4294967295
[ 5345.268079] Lustre: 89417:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
[ 5346.276087] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 11:52:51 (1780329171)
[ 5347.703448] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 5348.826678] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 11:52:53 (1780329173)
[ 5349.861284] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 5350.786600] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 11:52:55 (1780329175)
[ 5371.804178] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 11:53:16 (1780329196)
[ 5378.821499] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5401.578836] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 192.168.202.122@tcp) was lost; in progress operations using this service will fail
[ 5401.587502] LustreError: Skipped 2 previous similar messages
[ 5401.593095] Lustre: Evicted from MGS (at 192.168.202.122@tcp) after server handle changed from 0x555c39731cf9404c to 0x555c39731cf959f2
[ 5401.598246] Lustre: Skipped 2 previous similar messages
[ 5405.870519] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5406.834733] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5412.732650] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 11:53:58 (1780329238)
[ 5413.857338] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 5414.783705] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 11:54:00 (1780329240)
[ 5428.229467] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 11:54:13 (1780329253)
[ 5442.535934] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 192.168.202.122@tcp) was lost; in progress operations using this service will fail
[ 5442.544063] LustreError: Skipped 1 previous similar message
[ 5442.562445] Lustre: Evicted from MGS (at 192.168.202.122@tcp) after server handle changed from 0x555c39731cf95cae to 0x555c39731cf96035
[ 5442.572034] Lustre: Skipped 1 previous similar message
[ 5451.700872] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 11:54:36 (1780329276)
[ 5484.556138] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5486.539377] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5565.416725] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 192.168.202.122@tcp) was lost; in progress operations using this service will fail
[ 5565.423479] Lustre: lustre-MDT0000-mdc-ffff895485170000: Connection to lustre-MDT0000 (at 192.168.202.122@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5565.430530] Lustre: Skipped 10 previous similar messages
[ 5565.431402] Lustre: Evicted from MGS (at 192.168.202.122@tcp) after server handle changed from 0x555c39731cf96035 to 0x555c39731cf96e82
[ 5565.439785] Lustre: MGC192.168.202.122@tcp: Connection restored to 192.168.202.122@tcp (at 192.168.202.122@tcp)
[ 5565.444595] Lustre: Skipped 10 previous similar messages
[ 5571.338853] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5572.135876] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5597.575030] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5598.490178] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5674.308516] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 11:58:19 (1780329499)
[ 5708.105756] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5710.266350] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5989.915410] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 12:03:34 (1780329814)
[ 6133.731596] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 192.168.202.122@tcp) was lost; in progress operations using this service will fail
[ 6133.742773] LustreError: Skipped 1 previous similar message
[ 6133.794604] Lustre: Evicted from MGS (at 192.168.202.122@tcp) after server handle changed from 0x555c39731cf971ae to 0x555c39731d09143b
[ 6133.811439] Lustre: Skipped 1 previous similar message
[ 6177.882244] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 12:06:41 (1780330001)
[ 6179.778353] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 6181.498355] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 12:06:46 (1780330006)
[ 6184.074589] LustreError: lustre-MDT0000-mdc-ffff895485170000: operation ldlm_enqueue to node 192.168.202.122@tcp failed: rc = -107
[ 6184.087072] LustreError: Skipped 204 previous similar messages
[ 6184.092491] Lustre: lustre-MDT0000-mdc-ffff895485170000: Connection to lustre-MDT0000 (at 192.168.202.122@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6184.114461] Lustre: Skipped 3 previous similar messages
[ 6184.137171] LustreError: lustre-MDT0000-mdc-ffff895485170000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6184.174190] Lustre: lustre-MDT0000-mdc-ffff895485170000: Connection restored to 192.168.202.122@tcp (at 192.168.202.122@tcp)
[ 6184.204534] Lustre: Skipped 5 previous similar messages
[ 6193.700086] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 12:06:57 (1780330017)
[ 6360.047388] LustreError: lustre-OST0000-osc-ffff895485170000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6362.822874] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 12:09:46 (1780330186)
[ 6387.680054] Lustre: 2371:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780330193/real 1780330193]  req@ffff8954ae651f80 x1866804988673664/t0(0) o4->lustre-OST0000-osc-ffff895485170000@192.168.202.122@tcp:6/4 lens 4584/448 e 0 to 1 dl 1780330213 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 6387.714989] Lustre: 2371:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
[ 6404.876858] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 12:10:29 (1780330229)
[ 6406.247218] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 6407.677365] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 12:10:32 (1780330232)
[ 6409.486870] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 6411.579736] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 12:10:35 (1780330235)
[ 6445.519809] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 12:11:09 (1780330269)
[ 6479.854477] LustreError: MGC192.168.202.122@tcp: Connection to MGS (at 192.168.202.122@tcp) was lost; in progress operations using this service will fail
[ 6479.887434] Lustre: Evicted from MGS (at 192.168.202.122@tcp) after server handle changed from 0x555c39731d09143b to 0x555c39731d093c74
[ 6479.901960] Lustre: Skipped 1 previous similar message
[ 6493.813178] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 12:11:58 (1780330318)
[ 6495.043770] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 6496.211354] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 12:12:01 (1780330321)
[ 6497.492328] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 6498.827621] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 12:12:03 (1780330323)
[ 6513.123818] LustreError: 105889:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff895485170000: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 6513.138824] LustreError: 105889:0:(lov_obd.c:792:lov_cleanup()) Skipped 1 previous similar message
[ 6513.225722] Lustre: Unmounted lustre-client
[ 6514.714154] Lustre: Mounted lustre-client
[ 6517.182728] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6543.990978] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2