[ 1732.447194] LustreError: lustre-OST0000-osc-ffff94fbd9f3e800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1732.467331] Lustre: 2358:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.117@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1741.326725] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 22:15:25 (1789265725)
[ 1745.697287] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1748.886678] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 22:15:32 (1789265732)
[ 1750.991147] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1753.523439] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 22:15:37 (1789265737)
[ 1757.438917] LustreError: lustre-MDT0000-mdc-ffff94fbd9f3e800: operation mds_close to node 192.168.201.117@tcp failed: rc = -19
[ 1757.448721] LustreError: Skipped 2 previous similar messages
[ 1774.434501] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 192.168.201.117@tcp) was lost; in progress operations using this service will fail
[ 1783.614568] Lustre: Evicted from MGS (at 192.168.201.117@tcp) after server handle changed from 0xeb68ec3483f61e4d to 0xeb68ec3483f63449
[ 1785.457647] Lustre: 15905:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.117@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 1898.040632] LustreError: lustre-MDT0000-mdc-ffff94fbd9f3e800: operation mds_close to node 192.168.201.117@tcp failed: rc = -19
[ 1915.871793] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 192.168.201.117@tcp) was lost; in progress operations using this service will fail
[ 1926.057473] Lustre: Evicted from MGS (at 192.168.201.117@tcp) after server handle changed from 0xeb68ec3483f63449 to 0xeb68ec3483f80b8e
[ 1943.868267] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 22:18:48 (1789265928)
[ 1944.347972] Lustre: *** cfs_fail_loc=305, val=0***
[ 1944.352614] Lustre: Skipped 4 previous similar messages
[ 1992.160971] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 192.168.201.117@tcp) was lost; in progress operations using this service will fail
[ 2002.409701] Lustre: Evicted from MGS (at 192.168.201.117@tcp) after server handle changed from 0xeb68ec3483f80b8e to 0xeb68ec3483f8103b
[ 2002.423302] Lustre: MGC192.168.201.117@tcp: Connection restored to 192.168.201.117@tcp (at 192.168.201.117@tcp)
[ 2002.438258] Lustre: Skipped 13 previous similar messages
[ 2006.210383] Lustre: lustre-MDT0000-mdc-ffff94fbd9f3e800: Connection to lustre-MDT0000 (at 192.168.201.117@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2006.226711] Lustre: Skipped 10 previous similar messages
[ 2023.130895] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2024.802333] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2034.957496] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 22:20:19 (1789266019)
[ 2048.488980] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 192.168.201.117@tcp) was lost; in progress operations using this service will fail
[ 2048.539641] Lustre: Evicted from MGS (at 192.168.201.117@tcp) after server handle changed from 0xeb68ec3483f8103b to 0xeb68ec3483f813f3
[ 2057.908339] LustreError: lustre-MDT0000-mdc-ffff94fbd9f3e800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2074.662522] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 22:20:59 (1789266059)
[ 2112.785564] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 22:21:37 (1789266097)
[ 2126.665880] LustreError: lustre-MDT0000-mdc-ffff94fbd9f3e800: operation mds_reint to node 192.168.201.117@tcp failed: rc = -19
[ 2126.672159] LustreError: Skipped 2 previous similar messages
[ 2144.735895] Lustre: 2358:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789266115/real 1789266115]  req@ffff94fbc880b480 x1876179333313024/t0(0) o400->MGC192.168.201.117@tcp@192.168.201.117@tcp:26/25 lens 224/224 e 0 to 1 dl 1789266131 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2144.777338] Lustre: 2358:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 19 previous similar messages
[ 2144.787033] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 192.168.201.117@tcp) was lost; in progress operations using this service will fail
[ 2155.012473] Lustre: Evicted from MGS (at 192.168.201.117@tcp) after server handle changed from 0xeb68ec3483f813f3 to 0xeb68ec3483f85cbd
[ 2176.564172] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2180.291872] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2267.167410] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 192.168.201.117@tcp) was lost; in progress operations using this service will fail
[ 2277.423106] Lustre: Evicted from MGS (at 192.168.201.117@tcp) after server handle changed from 0xeb68ec3483f85cbd to 0xeb68ec3483f9b2dd
[ 2297.213436] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2299.888072] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2382.815986] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 192.168.201.117@tcp) was lost; in progress operations using this service will fail
[ 2393.028950] Lustre: Evicted from MGS (at 192.168.201.117@tcp) after server handle changed from 0xeb68ec3483f9b2dd to 0xeb68ec3483fb2d69
[ 2394.874881] Lustre: 15905:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.117@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2416.253032] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2419.323443] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2449.365412] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 22:27:14 (1789266434)
[ 2454.959182] LustreError: lustre-MDT0000-mdc-ffff94fbd9f3e800: operation ldlm_enqueue to node 192.168.201.117@tcp failed: rc = -19
[ 2454.980753] LustreError: Skipped 2 previous similar messages
[ 2474.975293] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 192.168.201.117@tcp) was lost; in progress operations using this service will fail
[ 2475.000462] Lustre: Evicted from MGS (at 192.168.201.117@tcp) after server handle changed from 0xeb68ec3483fb2d69 to 0xeb68ec3483fc0a60
[ 2476.586341] Lustre: 15905:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.117@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2493.017907] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2534.813554] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2580.548807] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2617.311796] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 192.168.201.117@tcp) was lost; in progress operations using this service will fail
[ 2617.342313] LustreError: Skipped 2 previous similar messages
[ 2627.575353] Lustre: Evicted from MGS (at 192.168.201.117@tcp) after server handle changed from 0xeb68ec3483fc4386 to 0xeb68ec3483fc8b7e
[ 2627.595867] Lustre: Skipped 2 previous similar messages
[ 2627.606992] Lustre: MGC192.168.201.117@tcp: Connection restored to 192.168.201.117@tcp (at 192.168.201.117@tcp)
[ 2627.614613] Lustre: Skipped 15 previous similar messages
[ 2638.438363] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2662.789592] Lustre: lustre-MDT0000-mdc-ffff94fbd9f3e800: Connection to lustre-MDT0000 (at 192.168.201.117@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2662.817359] Lustre: Skipped 8 previous similar messages
[ 2693.031252] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2760.491815] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2812.895892] Lustre: 2360:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789266783/real 1789266783]  req@ffff94fbc8816300 x1876179338233088/t0(0) o400->MGC192.168.201.117@tcp@192.168.201.117@tcp:26/25 lens 224/224 e 0 to 1 dl 1789266799 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2812.946212] Lustre: 2360:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
[ 2861.522287] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 22:34:06 (1789266846)
[ 2923.902622] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 2927.201456] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3208.124823] LustreError: lustre-OST0000-osc-ffff94fbd9f3e800: operation ldlm_enqueue to node 192.168.201.117@tcp failed: rc = -19
[ 3208.138520] LustreError: Skipped 12 previous similar messages
[ 3231.620900] Lustre: lustre-OST0000-osc-ffff94fbd9f3e800: Connection restored to 192.168.201.117@tcp (at 192.168.201.117@tcp)
[ 3231.633944] Lustre: Skipped 7 previous similar messages
[ 3253.704145] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3257.219115] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3538.679918] Lustre: lustre-OST0000-osc-ffff94fbd9f3e800: Connection to lustre-OST0000 (at 192.168.201.117@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3538.691810] Lustre: Skipped 4 previous similar messages
[ 3587.839173] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3590.413980] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3836.184675] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 22:50:21 (1789267821)
[ 3854.303965] Lustre: 50554:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789267824/real 1789267824]  req@ffff94fbd11b9880 x1876179353603584/t0(0) o101->lustre-MDT0000-mdc-ffff94fbd9f3e800@192.168.201.117@tcp:12/10 lens 576/1152 e 0 to 1 dl 1789267840 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3854.381330] Lustre: lustre-MDT0000-mdc-ffff94fbd9f3e800: Connection restored to 192.168.201.117@tcp (at 192.168.201.117@tcp)
[ 3854.394069] Lustre: Skipped 1 previous similar message
[ 3863.600240] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 22:50:48 (1789267848)
[ 3889.646939] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 22:51:14 (1789267874)
[ 3917.729526] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 22:51:42 (1789267902)
[ 3918.459095] Lustre: Mounted lustre-client - version 2.17.57_104_g543d5f9
[ 3950.367444] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 192.168.201.117@tcp) was lost; in progress operations using this service will fail
[ 3950.398054] LustreError: Skipped 3 previous similar messages
[ 3959.800147] Lustre: Evicted from MGS (at 192.168.201.117@tcp) after server handle changed from 0xeb68ec3483fe23a0 to 0xeb68ec3484102543
[ 3959.815462] Lustre: Skipped 3 previous similar messages
[ 3961.076769] Lustre: 15905:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.117@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 3988.860409] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 3991.372398] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3993.409243] Lustre: Unmounted lustre-client
[ 3999.826977] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 22:53:05 (1789267985)
[ 4154.335262] Lustre: lustre-OST0000-osc-ffff94fbd9f3e800: Connection to lustre-OST0000 (at 192.168.201.117@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4154.347378] Lustre: Skipped 13 previous similar messages
[ 4296.982661] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 22:58:02 (1789268282)
[ 4348.399587] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 22:58:53 (1789268333)
[ 4351.430207] Lustre: Unmounted lustre-client
[ 4376.032863] Lustre: Mounted lustre-client - version 2.17.57_104_g543d5f9
[ 4383.124138] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 22:59:28 (1789268368)
[ 4383.513729] LustreError: 56485:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4384.559170] LustreError: 56485:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4384.750420] Lustre: *** cfs_fail_loc=305, val=0***
[ 4406.806438] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 22:59:51 (1789268391)
[ 4407.347313] Lustre: Mounted lustre-client - version 2.17.57_104_g543d5f9
[ 4409.442740] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4419.741643] Lustre: Unmounted lustre-client
[ 4427.377233] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 23:00:12 (1789268412)
[ 4525.221721] LustreError: lustre-MDT0000-mdc-ffff94fbd0810800: operation mds_reint to node 192.168.201.117@tcp failed: rc = -107
[ 4525.230887] LustreError: Skipped 1 previous similar message
[ 4547.551302] Lustre: 2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789268517/real 1789268517]  req@ffff94fbc5a31880 x1876179356323712/t0(0) o400->MGC192.168.201.117@tcp@192.168.201.117@tcp:26/25 lens 224/224 e 0 to 1 dl 1789268533 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4547.602898] Lustre: 2359:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 154 previous similar messages
[ 4547.620726] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 192.168.201.117@tcp) was lost; in progress operations using this service will fail
[ 4547.646927] Lustre: Evicted from MGS (at 192.168.201.117@tcp) after server handle changed from 0xeb68ec3484103d99 to 0xeb68ec348412eb22
[ 4547.665346] Lustre: MGC192.168.201.117@tcp: Connection restored to 192.168.201.117@tcp (at 192.168.201.117@tcp)
[ 4547.672446] Lustre: Skipped 24 previous similar messages
[ 4549.550293] Lustre: 55883:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.117@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 4739.991341] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 23:05:24 (1789268724)
[ 4745.828613] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4760.052732] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 192.168.201.117@tcp) was lost; in progress operations using this service will fail
[ 4760.092795] Lustre: Evicted from MGS (at 192.168.201.117@tcp) after server handle changed from 0xeb68ec348412eb22 to 0xeb68ec348417c1ff
[ 4770.388318] LustreError: lustre-MDT0000-mdc-ffff94fbd0810800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4779.540739] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 23:06:04 (1789268764)
[ 4780.032646] Lustre: Mounted lustre-client - version 2.17.57_104_g543d5f9
[ 4787.567673] LustreError: lustre-OST0000-osc-ffff94fbd0810800: operation ldlm_enqueue to node 192.168.201.117@tcp failed: rc = -107
[ 4787.573541] Lustre: lustre-OST0000-osc-ffff94fbd0810800: Connection to lustre-OST0000 (at 192.168.201.117@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4787.583666] Lustre: Skipped 12 previous similar messages
[ 4787.597636] LustreError: lustre-OST0000-osc-ffff94fbd0810800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4799.685922] Lustre: Unmounted lustre-client
[ 4806.313513] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 23:06:31 (1789268791)
[ 4813.092820] LustreError: lustre-MDT0000-mdc-ffff94fbd0810800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4813.106259] LustreError: 60892:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4813.106818] LustreError: 60976:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff94fbd0810800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff94fbc5ae3700) refcount nonzero (2) after lock cleanup; forcing cleanup.
[ 4819.567812] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 23:06:44 (1789268804)
[ 4819.848023] LustreError: 61597:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4823.879458] LustreError: lustre-MDT0000-mdc-ffff94fbd0810800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4823.893503] LustreError: 61613:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4823.896428] LustreError: 61597:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=959
[ 4823.911709] LustreError: 61597:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting
[ 4823.967986] LustreError: 61618:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4825.085233] LustreError: 61625:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4825.105023] LustreError: 61625:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4827.464315] LustreError: 61647:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4827.472846] LustreError: 61647:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4831.958969] LustreError: 61691:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4831.971817] LustreError: 61691:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4841.574710] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 23:07:06 (1789268826)
[ 4880.150400] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4882.343594] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4894.552918] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 23:07:59 (1789268879)
[ 4934.485501] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4936.605137] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4946.450621] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 23:08:51 (1789268931)
[ 5011.017571] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5012.575191] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5020.586402] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 23:10:05 (1789269005)
[ 5061.814053] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5064.362866] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5071.561845] Lustre: Unmounted lustre-client
[ 5129.031296] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5131.757420] Lustre: Mounted lustre-client - version 2.17.57_104_g543d5f9
[ 5137.702524] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 23:12:02 (1789269122)
[ 5140.002815] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5141.740388] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 23:12:06 (1789269126)
[ 5156.418138] Lustre: lustre-OST0000-osc-ffff94fbc2e4d800: Connection restored to 192.168.201.117@tcp (at 192.168.201.117@tcp)
[ 5156.428315] Lustre: Skipped 12 previous similar messages
[ 5173.266939] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 23:12:38 (1789269158)
[ 5174.586503] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5176.114500] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 23:12:41 (1789269161)
[ 5177.386432] Lustre: *** cfs_fail_loc=805, val=0***
[ 5177.452957] Lustre: Mounted lustre-client - version 2.17.57_104_g543d5f9
[ 5181.904147] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5202.847257] Lustre: 2361:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789269173/real 1789269173]  req@ffff94fbd8f51180 x1876179358399360/t0(0) o400->MGC192.168.201.117@tcp@192.168.201.117@tcp:26/25 lens 224/224 e 0 to 1 dl 1789269189 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5202.892241] Lustre: 2361:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5202.900116] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 192.168.201.117@tcp) was lost; in progress operations using this service will fail
[ 5202.948326] Lustre: Evicted from MGS (at 192.168.201.117@tcp) after server handle changed from 0xeb68ec348417cc9b to 0xeb68ec348417ceb6
[ 5204.343543] Lustre: 68254:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.117@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 5221.623251] Lustre: Unmounted lustre-client
[ 5229.513822] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 23:13:34 (1789269214)
[ 5255.789318] Lustre: 68254:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.117@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 5272.147160] LustreError: 2357:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff94fbd86bd880 x1876179358411904/t94489280517(94489280517) o101->lustre-MDT0000-mdc-ffff94fbc2e4d800@192.168.201.117@tcp:12/10 lens 648/608 e 0 to 0 dl 1789269274 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5280.274338] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5282.321758] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5291.802859] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 23:14:37 (1789269277)
[ 5293.397347] LustreError: lustre-OST0000-osc-ffff94fbc2e4d800: operation ost_write to node 192.168.201.117@tcp failed: rc = -107
[ 5293.417994] LustreError: Skipped 2 previous similar messages
[ 5293.435969] LustreError: lustre-OST0000-osc-ffff94fbc2e4d800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5293.459659] Lustre: 2360:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.117@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5293.483854] LustreError: 72792:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff94fbc2e4d800: namespace resource [0x240000400:0x37e2:0x0].0x0 (ffff94fbc5a6bd00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5301.920692] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 23:14:46 (1789269286)
[ 5304.046337] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5305.908670] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 23:14:50 (1789269290)
[ 5307.361391] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5309.263733] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 23:14:54 (1789269294)
[ 5310.553816] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5312.248758] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 23:14:57 (1789269297)
[ 5313.369383] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5314.933857] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 23:15:00 (1789269300)
[ 5316.408119] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5318.719626] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 23:15:03 (1789269303)
[ 5320.423607] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5322.360345] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 23:15:07 (1789269307)
[ 5323.851493] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5325.508196] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 23:15:10 (1789269310)
[ 5327.413865] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5329.134700] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 23:15:14 (1789269314)
[ 5330.904471] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5333.506953] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 23:15:18 (1789269318)
[ 5335.333429] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5337.461739] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 23:15:22 (1789269322)
[ 5339.081701] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5341.251217] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 23:15:26 (1789269326)
[ 5344.731144] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5346.527553] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 23:15:31 (1789269331)
[ 5369.334611] Lustre: 68254:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.117@tcp: IR log lustre-cliir failed, not fatal: rc = -5
[ 5380.625975] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 23:16:05 (1789269365)
[ 5412.199579] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 23:16:36 (1789269396)
[ 5412.738432] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5423.546419] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 23:16:48 (1789269408)
[ 5424.123362] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5434.753855] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 23:17:00 (1789269420)
[ 5435.378553] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5444.076891] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 23:17:09 (1789269429)
[ 5444.601222] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5453.938587] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 23:17:18 (1789269438)
[ 5454.572143] Lustre: *** cfs_fail_loc=510, val=3***
[ 5466.753353] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 23:17:31 (1789269451)
[ 5467.453157] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5525.983420] Lustre: lustre-OST0001-osc-ffff94fbc2e4d800: Connection to lustre-OST0001 (at 192.168.201.117@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5525.993515] Lustre: Skipped 16 previous similar messages
[ 5532.929286] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 23:18:37 (1789269517)
[ 5533.278331] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5600.481909] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 23:19:45 (1789269585)
[ 5601.120216] LustreError: 83009:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5604.582403] LustreError: lustre-MDT0000-mdc-ffff94fbc2e4d800: operation ldlm_enqueue to node 192.168.201.117@tcp failed: rc = -107
[ 5604.608823] LustreError: lustre-MDT0000-mdc-ffff94fbc2e4d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5604.656392] LustreError: 83023:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff94fbc2e4d800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff94fbc5a6b300) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5605.199148] LustreError: 83009:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5607.360942] LustreError: 83039:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5610.730266] LustreError: lustre-MDT0000-mdc-ffff94fbc2e4d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5610.753499] LustreError: 83054:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff94fbc2e4d800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff94fbc5a6b300) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5610.774273] LustreError: 83054:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5611.447161] LustreError: 83039:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5618.629475] LustreError: lustre-MDT0000-mdc-ffff94fbc2e4d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5619.060892] LustreError: 83095:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5622.410240] LustreError: lustre-MDT0000-mdc-ffff94fbc2e4d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5622.495539] LustreError: 83115:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5622.502437] LustreError: 83115:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5623.135255] LustreError: 83095:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5623.158213] LustreError: 83095:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff94fbc2e4d800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5623.618917] LustreError: 83121:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5623.649737] LustreError: 83121:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5626.203360] LustreError: 83143:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5626.222717] LustreError: 83143:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5629.987775] LustreError: 83170:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5634.065873] LustreError: lustre-MDT0000-mdc-ffff94fbc2e4d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5634.075842] LustreError: 83170:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5634.082604] LustreError: 83185:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff94fbc2e4d800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff94fbc5a6b400) refcount nonzero (2) after lock cleanup; forcing cleanup.
[ 5634.114923] LustreError: 83185:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5641.164906] LustreError: lustre-MDT0000-mdc-ffff94fbc2e4d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5641.594249] LustreError: 83226:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5644.527988] LustreError: lustre-MDT0000-mdc-ffff94fbc2e4d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5644.537043] LustreError: 83241:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff94fbc2e4d800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff94fbc5b0fb00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5645.663383] LustreError: 83226:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5648.737389] LustreError: lustre-MDT0000-mdc-ffff94fbc2e4d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5648.802954] LustreError: 83274:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5648.807954] LustreError: 83274:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5649.927647] LustreError: 83254:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff94fbc2e4d800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5657.827561] LustreError: 83326:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5657.841824] LustreError: 83326:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5661.150899] LustreError: lustre-MDT0000-mdc-ffff94fbc2e4d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5661.216720] LustreError: 83348:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5661.223016] LustreError: 83348:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5661.919536] LustreError: 83326:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5661.927260] LustreError: 83326:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5661.936586] LustreError: 83326:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff94fbc2e4d800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5672.831561] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 23:20:57 (1789269657)
[ 5691.499039] LustreError: 68251:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5691.511081] LustreError: 68251:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5695.575239] LustreError: 68251:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5695.585073] LustreError: 68251:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5702.618796] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 23:21:27 (1789269687)
[ 5755.078288] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 23:22:20 (1789269740)
[ 5813.216700] Lustre: 85507:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789269744/real 1789269744]  req@ffff94fbd0848e00 x1876179358542720/t0(0) o101->lustre-MDT0000-mdc-ffff94fbc2e4d800@192.168.201.117@tcp:12/10 lens 576/1152 e 0 to 1 dl 1789269799 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'stat.0' uid:0 gid:0 projid:0
[ 5813.257857] Lustre: 85507:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 15 previous similar messages
[ 5813.292893] Lustre: lustre-MDT0000-mdc-ffff94fbc2e4d800: Connection restored to 192.168.201.117@tcp (at 192.168.201.117@tcp)
[ 5813.301724] Lustre: Skipped 19 previous similar messages
[ 5821.318559] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 23:23:26 (1789269806)
[ 5857.608187] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 23:24:02 (1789269842)
[ 5858.190535] Lustre: Mounted lustre-client - version 2.17.57_104_g543d5f9
[ 5982.442586] Lustre: Unmounted lustre-client
[ 5991.302477] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 23:26:15 (1789269975)
[ 5992.253173] LustreError: 87655:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5995.220729] LustreError: lustre-OST0000-osc-ffff94fbc2e4d800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5995.233474] LustreError: 87742:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff94fbc2e4d800: namespace resource [0x240000400:0x380d:0x0].0x0 (ffff94fbc5a6b800) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5995.261622] LustreError: 87742:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5995.472956] LustreError: 87655:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5995.480473] Lustre: 2358:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.117@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 6003.166225] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 23:26:27 (1789269987)
[ 6067.156864] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 23:27:32 (1789270052)
[ 6068.454772] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 6069.868875] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 23:27:35 (1789270055)
[ 6126.559555] Lustre: lustre-MDT0000-mdc-ffff94fbc2e4d800: Connection to lustre-MDT0000 (at 192.168.201.117@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6126.577177] Lustre: Skipped 15 previous similar messages
[ 6134.655958] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6136.074480] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 23:28:41 (1789270121)
[ 6194.934436] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 23:29:39 (1789270179)
[ 6197.277784] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6199.279776] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 23:29:43 (1789270183)
[ 6200.827903] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6202.891118] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 23:29:47 (1789270187)
[ 6244.087244] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 23:30:29 (1789270229)
[ 6258.872292] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6286.304493] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 192.168.201.117@tcp) was lost; in progress operations using this service will fail
[ 6286.315870] LustreError: Skipped 3 previous similar messages
[ 6296.581313] Lustre: Evicted from MGS (at 192.168.201.117@tcp) after server handle changed from 0xeb68ec348417db0b to 0xeb68ec348417f43a
[ 6296.595977] Lustre: Skipped 3 previous similar messages
[ 6310.433704] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6312.574759] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6324.063321] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 23:31:48 (1789270308)
[ 6326.887378] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6328.975618] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 23:31:53 (1789270313)
[ 6342.633487] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 192.168.201.117@tcp) was lost; in progress operations using this service will fail
[ 6342.653816] Lustre: Evicted from MGS (at 192.168.201.117@tcp) after server handle changed from 0xeb68ec348417f43a to 0xeb68ec348417f7f9
[ 6354.274221] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 23:32:19 (1789270339)
[ 6405.190944] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 23:33:10 (1789270390)
[ 6413.647527] LustreError: lustre-OST0000-osc-ffff94fbc2e4d800: operation ost_setattr to node 192.168.201.117@tcp failed: rc = -19
[ 6413.675643] LustreError: Skipped 9 previous similar messages
[ 6417.574215] Lustre: 2360:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789270349/real 1789270349]  req@ffff94fbd8f52d80 x1876179358637184/t0(0) o400->lustre-MDT0000-mdc-ffff94fbc2e4d800@192.168.201.117@tcp:12/10 lens 224/224 e 0 to 1 dl 1789270404 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6417.591224] Lustre: 2360:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 14 previous similar messages
[ 6479.003813] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6484.172472] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6572.009413] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 192.168.201.117@tcp) was lost; in progress operations using this service will fail
[ 6572.028549] LustreError: Skipped 1 previous similar message
[ 6572.041191] Lustre: Evicted from MGS (at 192.168.201.117@tcp) after server handle changed from 0xeb68ec348417f9ff to 0xeb68ec348418084c
[ 6572.052718] Lustre: Skipped 1 previous similar message
[ 6572.070690] Lustre: MGC192.168.201.117@tcp: Connection restored to 192.168.201.117@tcp (at 192.168.201.117@tcp)
[ 6572.078708] Lustre: Skipped 10 previous similar messages
[ 6573.359407] Lustre: 68254:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.117@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6598.809819] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6600.628310] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6647.484939] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6649.432452] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6790.139414] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 23:39:34 (1789270774)
[ 6801.131493] Lustre: lustre-OST0000-osc-ffff94fbc2e4d800: Connection to lustre-OST0000 (at 192.168.201.117@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6801.142932] Lustre: Skipped 7 previous similar messages
[ 6857.873744] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6862.329790] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 7123.072743] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 23:45:08 (1789271108)
[ 7284.709703] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 192.168.201.117@tcp) was lost; in progress operations using this service will fail
[ 7284.727034] LustreError: Skipped 1 previous similar message
[ 7284.739637] Lustre: Evicted from MGS (at 192.168.201.117@tcp) after server handle changed from 0xeb68ec3484180cf9 to 0xeb68ec348427aefa
[ 7284.748564] Lustre: Skipped 1 previous similar message
[ 7284.763568] Lustre: MGC192.168.201.117@tcp: Connection restored to 192.168.201.117@tcp (at 192.168.201.117@tcp)
[ 7284.775271] Lustre: Skipped 3 previous similar messages
[ 7311.327202] Lustre: 2358:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789271260/real 1789271260]  req@ffff94fbc5be1880 x1876179376275584/t0(0) o400->lustre-MDT0000-mdc-ffff94fbc2e4d800@192.168.201.117@tcp:12/10 lens 224/224 e 0 to 1 dl 1789271297 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7311.362490] Lustre: 2358:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 13 previous similar messages
[ 7322.760461] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 23:48:27 (1789271307)
[ 7324.419204] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7327.101781] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 23:48:31 (1789271311)
[ 7329.420578] LustreError: lustre-MDT0000-mdc-ffff94fbc2e4d800: operation ldlm_enqueue to node 192.168.201.117@tcp failed: rc = -107
[ 7329.429631] LustreError: Skipped 40 previous similar messages
[ 7329.452708] LustreError: lustre-MDT0000-mdc-ffff94fbc2e4d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7339.051931] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 23:48:43 (1789271323)
[ 7506.418059] LustreError: lustre-OST0000-osc-ffff94fbc2e4d800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7508.334140] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 23:51:33 (1789271493)
[ 7534.559269] Lustre: lustre-OST0000-osc-ffff94fbc2e4d800: Connection to lustre-OST0000 (at 192.168.201.117@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7534.574546] Lustre: Skipped 3 previous similar messages
[ 7553.032578] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 23:52:18 (1789271538)
[ 7554.973189] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7557.243400] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 23:52:22 (1789271542)
[ 7559.070378] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7561.417922] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 23:52:26 (1789271546)
[ 7597.454955] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 23:53:02 (1789271582)
[ 7636.967140] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 192.168.201.117@tcp) was lost; in progress operations using this service will fail
[ 7636.987856] Lustre: Evicted from MGS (at 192.168.201.117@tcp) after server handle changed from 0xeb68ec348427aefa to 0xeb68ec348427d47e
[ 7636.994988] Lustre: Skipped 1 previous similar message
[ 7656.520287] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 23:54:01 (1789271641)
[ 7658.664578] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7660.800620] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 23:54:05 (1789271645)
[ 7662.603740] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7665.201797] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 23:54:09 (1789271649)
[ 7676.791591] Lustre: Unmounted lustre-client
[ 7679.427206] Lustre: Mounted lustre-client - version 2.17.57_104_g543d5f9
[ 7682.953974] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7727.069359] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2