[ 1960.312199] LustreError: lustre-OST0000-osc-ffff983344f86800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1960.332915] Lustre: 2363:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.111@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1968.561168] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 00:45:58 (1788583558)
[ 1970.914044] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1972.740738] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 00:46:02 (1788583562)
[ 1974.659258] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1976.167968] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 00:46:06 (1788583566)
[ 1997.791487] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 192.168.201.111@tcp) was lost; in progress operations using this service will fail
[ 2007.999887] Lustre: Evicted from MGS (at 192.168.201.111@tcp) after server handle changed from 0xc627f82fcc2b3896 to 0xc627f82fcc2b5457
[ 2106.026762] LustreError: lustre-MDT0000-mdc-ffff983344f86800: operation mds_reint to node 192.168.201.111@tcp failed: rc = -19
[ 2106.034493] LustreError: Skipped 3 previous similar messages
[ 2124.768578] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 192.168.201.111@tcp) was lost; in progress operations using this service will fail
[ 2134.976419] Lustre: Evicted from MGS (at 192.168.201.111@tcp) after server handle changed from 0xc627f82fcc2b5457 to 0xc627f82fcc2cd4c4
[ 2155.017970] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 00:49:04 (1788583744)
[ 2155.505989] Lustre: *** cfs_fail_loc=305, val=0***
[ 2155.510293] Lustre: Skipped 2 previous similar messages
[ 2176.268372] LustreError: lustre-OST0000-osc-ffff983344f86800: operation ost_connect to node 192.168.201.111@tcp failed: rc = -75
[ 2176.280460] LustreError: Skipped 1 previous similar message
[ 2200.543640] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 192.168.201.111@tcp) was lost; in progress operations using this service will fail
[ 2210.813397] Lustre: Evicted from MGS (at 192.168.201.111@tcp) after server handle changed from 0xc627f82fcc2cd4c4 to 0xc627f82fcc2ce7d3
[ 2210.813435] Lustre: lustre-MDT0000-mdc-ffff983344f86800: Connection to lustre-MDT0000 (at 192.168.201.111@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2210.828838] Lustre: MGC192.168.201.111@tcp: Connection restored to 192.168.201.111@tcp (at 192.168.201.111@tcp)
[ 2210.846876] Lustre: Skipped 11 previous similar messages
[ 2210.865526] Lustre: Skipped 13 previous similar messages
[ 2225.338414] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2227.221242] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2235.362938] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 00:50:25 (1788583825)
[ 2251.757354] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 192.168.201.111@tcp) was lost; in progress operations using this service will fail
[ 2251.801181] Lustre: Evicted from MGS (at 192.168.201.111@tcp) after server handle changed from 0xc627f82fcc2ce7d3 to 0xc627f82fcc2cef27
[ 2251.878773] LustreError: lustre-MDT0000-mdc-ffff983344f86800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2275.169071] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 00:51:05 (1788583865)
[ 2310.216289] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 00:51:40 (1788583900)
[ 2323.690070] LustreError: lustre-MDT0000-mdc-ffff983344f86800: operation mds_reint to node 192.168.201.111@tcp failed: rc = -19
[ 2343.916604] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 192.168.201.111@tcp) was lost; in progress operations using this service will fail
[ 2343.946842] Lustre: Evicted from MGS (at 192.168.201.111@tcp) after server handle changed from 0xc627f82fcc2cef27 to 0xc627f82fcc2d3899
[ 2345.712111] Lustre: 15940:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.111@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2374.803617] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2377.388856] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2461.663388] Lustre: 2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788584037/real 1788584037]  req@ffff983344a31500 x1875463796626304/t0(0) o400->MGC192.168.201.111@tcp@192.168.201.111@tcp:26/25 lens 224/224 e 0 to 1 dl 1788584053 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2461.688049] Lustre: 2366:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 14 previous similar messages
[ 2461.698205] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 192.168.201.111@tcp) was lost; in progress operations using this service will fail
[ 2471.920853] Lustre: Evicted from MGS (at 192.168.201.111@tcp) after server handle changed from 0xc627f82fcc2d3899 to 0xc627f82fcc2ebc94
[ 2473.464964] Lustre: 15940:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.111@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2493.122238] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2495.267854] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2579.423261] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 192.168.201.111@tcp) was lost; in progress operations using this service will fail
[ 2588.592740] Lustre: Evicted from MGS (at 192.168.201.111@tcp) after server handle changed from 0xc627f82fcc2ebc94 to 0xc627f82fcc30692a
[ 2590.333732] Lustre: 15940:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.111@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2615.087602] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2618.151382] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2648.396372] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 00:57:18 (1788584238)
[ 2653.392453] LustreError: lustre-MDT0000-mdc-ffff983344f86800: operation ldlm_enqueue to node 192.168.201.111@tcp failed: rc = -19
[ 2653.406825] LustreError: Skipped 4 previous similar messages
[ 2670.559453] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 192.168.201.111@tcp) was lost; in progress operations using this service will fail
[ 2680.824142] Lustre: Evicted from MGS (at 192.168.201.111@tcp) after server handle changed from 0xc627f82fcc30692a to 0xc627f82fcc314683
[ 2687.104061] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2733.006975] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2770.930770] Lustre: 15940:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.111@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2784.961539] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2815.967712] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 192.168.201.111@tcp) was lost; in progress operations using this service will fail
[ 2815.978106] LustreError: Skipped 2 previous similar messages
[ 2826.161297] Lustre: Evicted from MGS (at 192.168.201.111@tcp) after server handle changed from 0xc627f82fcc318e89 to 0xc627f82fcc31f6b7
[ 2826.175903] Lustre: Skipped 2 previous similar messages
[ 2826.188379] Lustre: MGC192.168.201.111@tcp: Connection restored to 192.168.201.111@tcp (at 192.168.201.111@tcp)
[ 2826.195769] Lustre: Skipped 15 previous similar messages
[ 2827.243380] Lustre: 15940:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.111@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2842.144804] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2868.071667] Lustre: lustre-MDT0000-mdc-ffff983344f86800: Connection to lustre-MDT0000 (at 192.168.201.111@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2868.094036] Lustre: Skipped 9 previous similar messages
[ 2909.812566] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2982.054546] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 3084.399783] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 01:04:33 (1788584673)
[ 3147.512340] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3150.513649] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3431.166225] LustreError: lustre-OST0000-osc-ffff983344f86800: operation ldlm_enqueue to node 192.168.201.111@tcp failed: rc = -19
[ 3431.183761] LustreError: Skipped 8 previous similar messages
[ 3455.508177] Lustre: lustre-OST0000-osc-ffff983344f86800: Connection restored to 192.168.201.111@tcp (at 192.168.201.111@tcp)
[ 3455.527314] Lustre: Skipped 7 previous similar messages
[ 3483.897984] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3486.797279] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3762.844353] Lustre: lustre-OST0000-osc-ffff983344f86800: Connection to lustre-OST0000 (at 192.168.201.111@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3762.860892] Lustre: Skipped 4 previous similar messages
[ 3814.897835] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3819.370083] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4060.840409] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 01:20:50 (1788585650)
[ 4079.071516] Lustre: 50585:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788585654/real 1788585654]  req@ffff983344049180 x1875463814981248/t0(0) o101->lustre-MDT0000-mdc-ffff983344f86800@192.168.201.111@tcp:12/10 lens 576/1152 e 0 to 1 dl 1788585670 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 4079.123650] Lustre: 50585:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
[ 4079.177989] Lustre: lustre-MDT0000-mdc-ffff983344f86800: Connection restored to 192.168.201.111@tcp (at 192.168.201.111@tcp)
[ 4079.187205] Lustre: Skipped 1 previous similar message
[ 4089.807962] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 01:21:19 (1788585679)
[ 4115.685881] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 01:21:45 (1788585705)
[ 4141.166339] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 01:22:11 (1788585731)
[ 4141.738211] Lustre: Mounted lustre-client - version 2.17.57_104_g4a8f637
[ 4172.767278] Lustre: 2365:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788585748/real 1788585748]  req@ffff9833448b8e00 x1875463815000448/t0(0) o400->MGC192.168.201.111@tcp@192.168.201.111@tcp:26/25 lens 224/224 e 0 to 1 dl 1788585764 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4172.823119] Lustre: 2365:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 4172.838759] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 192.168.201.111@tcp) was lost; in progress operations using this service will fail
[ 4172.851293] LustreError: Skipped 3 previous similar messages
[ 4183.030300] Lustre: Evicted from MGS (at 192.168.201.111@tcp) after server handle changed from 0xc627f82fcc33e6bb to 0xc627f82fcc444442
[ 4183.037679] Lustre: Skipped 3 previous similar messages
[ 4190.747681] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4192.259540] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4194.640826] Lustre: Unmounted lustre-client
[ 4202.179263] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 01:23:12 (1788585792)
[ 4324.319462] Lustre: 2364:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788585899/real 1788585899]  req@ffff98334404aa00 x1875463815034240/t0(0) o4->lustre-OST0000-osc-ffff983344f86800@192.168.201.111@tcp:6/4 lens 488/448 e 0 to 1 dl 1788585915 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4324.390637] Lustre: 2364:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 61 previous similar messages
[ 4372.448528] Lustre: lustre-OST0000-osc-ffff983344f86800: Connection to lustre-OST0000 (at 192.168.201.111@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4372.461018] Lustre: Skipped 14 previous similar messages
[ 4502.614159] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 01:28:12 (1788586092)
[ 4552.379512] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 01:29:02 (1788586142)
[ 4555.030293] Lustre: Unmounted lustre-client
[ 4580.710857] Lustre: Mounted lustre-client - version 2.17.57_104_g4a8f637
[ 4587.532701] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 01:29:37 (1788586177)
[ 4588.030211] LustreError: 56511:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4589.092395] LustreError: 56511:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4589.390746] Lustre: *** cfs_fail_loc=305, val=0***
[ 4615.000925] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 01:30:04 (1788586204)
[ 4615.622793] Lustre: Mounted lustre-client - version 2.17.57_104_g4a8f637
[ 4619.121398] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4629.472213] Lustre: Unmounted lustre-client
[ 4637.115204] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 01:30:27 (1788586227)
[ 4745.022931] LustreError: lustre-MDT0000-mdc-ffff983344af7800: operation mds_reint to node 192.168.201.111@tcp failed: rc = -19
[ 4745.040662] LustreError: Skipped 1 previous similar message
[ 4765.155073] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 192.168.201.111@tcp) was lost; in progress operations using this service will fail
[ 4765.189237] Lustre: Evicted from MGS (at 192.168.201.111@tcp) after server handle changed from 0xc627f82fcc445d39 to 0xc627f82fcc470cac
[ 4765.200647] Lustre: MGC192.168.201.111@tcp: Connection restored to 192.168.201.111@tcp (at 192.168.201.111@tcp)
[ 4765.204839] Lustre: Skipped 24 previous similar messages
[ 4766.836486] Lustre: 55907:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.111@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 4956.703158] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 01:35:46 (1788586546)
[ 4963.000350] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4979.180080] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 192.168.201.111@tcp) was lost; in progress operations using this service will fail
[ 4979.195405] Lustre: lustre-MDT0000-mdc-ffff983344af7800: Connection to lustre-MDT0000 (at 192.168.201.111@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4979.209172] Lustre: Evicted from MGS (at 192.168.201.111@tcp) after server handle changed from 0xc627f82fcc470cac to 0xc627f82fcc4be087
[ 4979.219718] Lustre: Skipped 10 previous similar messages
[ 4979.340068] LustreError: lustre-MDT0000-mdc-ffff983344af7800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4996.300900] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 01:36:26 (1788586586)
[ 4996.862408] Lustre: Mounted lustre-client - version 2.17.57_104_g4a8f637
[ 5004.609391] LustreError: lustre-OST0000-osc-ffff983344af7800: operation ldlm_enqueue to node 192.168.201.111@tcp failed: rc = -107
[ 5004.632117] LustreError: lustre-OST0000-osc-ffff983344af7800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5004.645204] LustreError: 60239:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff983344af7800: namespace resource [0x240000400:0x3783:0x0].0x0 (ffff983346bd9d00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5014.495248] Lustre: 2366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788586589/real 1788586589]  req@ffff983348aeb100 x1875463819699584/t0(0) o103->lustre-OST0000-osc-ffff983343935000@192.168.201.111@tcp:17/18 lens 328/224 e 0 to 1 dl 1788586605 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295
[ 5014.555703] Lustre: 2366:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 76 previous similar messages
[ 5017.293868] Lustre: Unmounted lustre-client
[ 5026.426687] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 01:36:55 (1788586615)
[ 5033.815935] LustreError: lustre-MDT0000-mdc-ffff983344af7800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5033.834313] LustreError: 60912:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 5042.498295] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 01:37:12 (1788586632)
[ 5042.694681] LustreError: 61611:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 5047.775579] LustreError: 61611:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=0
[ 5047.787795] LustreError: 61611:0:(import.c:711:ptlrpc_connect_import_locked()) already connected
[ 5048.239995] LustreError: lustre-MDT0000-mdc-ffff983344af7800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5048.255104] LustreError: 61634:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 5048.363743] LustreError: 61639:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5049.501341] LustreError: 61645:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5049.517992] LustreError: 61645:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5051.817283] LustreError: 61668:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5051.823259] LustreError: 61668:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5056.522534] LustreError: 61712:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5056.529265] LustreError: 61712:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 5069.261968] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 01:37:38 (1788586658)
[ 5118.119199] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5120.046368] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5132.221413] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 01:38:41 (1788586721)
[ 5175.771806] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5177.561673] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5189.395672] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 01:39:39 (1788586779)
[ 5258.710529] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5260.264919] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5269.405443] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 01:40:59 (1788586859)
[ 5309.773222] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5311.695315] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5319.284067] Lustre: Unmounted lustre-client
[ 5380.638205] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5383.770865] Lustre: Mounted lustre-client - version 2.17.57_104_g4a8f637
[ 5391.522264] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 01:43:01 (1788586981)
[ 5394.288449] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5396.026185] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 01:43:05 (1788586985)
[ 5426.153917] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 01:43:36 (1788587016)
[ 5427.528608] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5429.450883] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 01:43:39 (1788587019)
[ 5431.282738] Lustre: *** cfs_fail_loc=805, val=0***
[ 5431.331597] Lustre: Mounted lustre-client - version 2.17.57_104_g4a8f637
[ 5436.970463] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5456.863459] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 192.168.201.111@tcp) was lost; in progress operations using this service will fail
[ 5467.119669] Lustre: Evicted from MGS (at 192.168.201.111@tcp) after server handle changed from 0xc627f82fcc4bea82 to 0xc627f82fcc4bee2c
[ 5467.139054] Lustre: MGC192.168.201.111@tcp: Connection restored to 192.168.201.111@tcp (at 192.168.201.111@tcp)
[ 5467.143365] LustreError: 2362:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9833465a3000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff983346bd9e00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5467.150989] Lustre: Skipped 12 previous similar messages
[ 5470.029225] Lustre: Unmounted lustre-client
[ 5478.830579] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 01:44:28 (1788587068)
[ 5505.143360] Lustre: 68286:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.111@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 5521.423198] LustreError: 2362:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff983344380380 x1875463819785728/t94489280517(94489280517) o101->lustre-MDT0000-mdc-ffff983348fab000@192.168.201.111@tcp:12/10 lens 648/608 e 0 to 0 dl 1788587128 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5527.905139] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5529.881183] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5539.565197] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 01:45:29 (1788587129)
[ 5541.515425] LustreError: lustre-OST0000-osc-ffff983348fab000: operation ost_write to node 192.168.201.111@tcp failed: rc = -107
[ 5541.526540] LustreError: Skipped 2 previous similar messages
[ 5541.549939] LustreError: lustre-OST0000-osc-ffff983348fab000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5541.559968] Lustre: 2366:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.111@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5541.595765] LustreError: 72828:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff983348fab000: namespace resource [0x240000400:0x37e2:0x0].0x0 (ffff9833470d7100) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5541.609572] LustreError: 72828:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5550.918371] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 01:45:40 (1788587140)
[ 5552.438401] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5554.073878] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 01:45:44 (1788587144)
[ 5555.459172] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5557.404973] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 01:45:47 (1788587147)
[ 5559.230811] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5561.388679] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 01:45:51 (1788587151)
[ 5562.877398] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5564.751304] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 01:45:55 (1788587155)
[ 5566.072996] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5567.757987] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 01:45:57 (1788587157)
[ 5569.408591] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5571.194089] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 01:46:01 (1788587161)
[ 5572.614833] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5574.358345] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 01:46:04 (1788587164)
[ 5576.092646] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5577.767578] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 01:46:08 (1788587168)
[ 5579.031942] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5580.769313] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 01:46:11 (1788587171)
[ 5582.413858] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5583.997650] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 01:46:14 (1788587174)
[ 5585.580771] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5586.997462] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 01:46:17 (1788587177)
[ 5589.659419] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5591.201457] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 01:46:21 (1788587181)
[ 5613.548212] Lustre: lustre-MDT0000-mdc-ffff983348fab000: Connection to lustre-MDT0000 (at 192.168.201.111@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5613.559039] Lustre: Skipped 14 previous similar messages
[ 5614.559425] Lustre: 2364:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788587189/real 1788587189]  req@ffff9833441d0a80 x1875463819802368/t0(0) o400->lustre-MDT0000-mdc-ffff983348fab000@192.168.201.111@tcp:12/10 lens 224/224 e 0 to 1 dl 1788587205 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5614.581299] Lustre: 2364:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 12 previous similar messages
[ 5625.423307] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 01:46:55 (1788587215)
[ 5654.779345] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 01:47:25 (1788587245)
[ 5655.284735] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5665.243663] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 01:47:35 (1788587255)
[ 5665.583119] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5675.286154] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 01:47:45 (1788587265)
[ 5675.749768] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5685.184512] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 01:47:55 (1788587275)
[ 5685.615711] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5694.438635] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 01:48:04 (1788587284)
[ 5694.835556] Lustre: *** cfs_fail_loc=510, val=3***
[ 5703.642882] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 01:48:13 (1788587293)
[ 5704.048187] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5771.067894] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 01:49:20 (1788587360)
[ 5771.868739] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5841.169388] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 01:50:30 (1788587430)
[ 5841.858470] LustreError: 83043:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5845.242942] LustreError: lustre-MDT0000-mdc-ffff983348fab000: operation ldlm_enqueue to node 192.168.201.111@tcp failed: rc = -107
[ 5845.262018] LustreError: lustre-MDT0000-mdc-ffff983348fab000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5845.288204] LustreError: 83058:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff983348fab000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff983346bd9800) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5845.873213] LustreError: 83043:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5848.021461] LustreError: 83073:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5851.368425] LustreError: lustre-MDT0000-mdc-ffff983348fab000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5851.384504] LustreError: 83088:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff983348fab000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff983346bd9e00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5851.425598] LustreError: 83088:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5852.087571] LustreError: 83073:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5859.416592] LustreError: lustre-MDT0000-mdc-ffff983348fab000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5859.467238] LustreError: 83117:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff983348fab000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff983346bd9300) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5859.486316] LustreError: 83117:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5859.947320] LustreError: 83130:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5862.961583] LustreError: lustre-MDT0000-mdc-ffff983348fab000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5863.028443] LustreError: 83149:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5863.035919] LustreError: 83149:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5863.953732] LustreError: 83130:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5863.966083] LustreError: 83130:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff983348fab000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5864.108906] LustreError: 83156:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5864.116266] LustreError: 83156:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5866.506115] LustreError: 83178:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5866.515834] LustreError: 83178:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5870.248821] LustreError: 83205:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5873.345477] LustreError: lustre-MDT0000-mdc-ffff983348fab000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5874.351188] LustreError: 83205:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5881.583368] LustreError: lustre-MDT0000-mdc-ffff983348fab000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5882.063208] LustreError: 83261:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5885.241158] LustreError: lustre-MDT0000-mdc-ffff983348fab000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5885.276966] LustreError: 83276:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff983348fab000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff983344107e00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5885.295855] LustreError: 83276:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 2 previous similar messages
[ 5886.127140] LustreError: 83261:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5889.270323] LustreError: lustre-MDT0000-mdc-ffff983348fab000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5889.358703] LustreError: 83308:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5889.366762] LustreError: 83308:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5890.306139] LustreError: 83289:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff983348fab000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5898.783324] LustreError: 83361:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5898.796024] LustreError: 83361:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5902.679700] LustreError: lustre-MDT0000-mdc-ffff983348fab000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5902.749451] LustreError: 83383:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5902.758207] LustreError: 83383:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5902.808389] LustreError: 83361:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5902.812231] LustreError: 83361:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5902.836293] LustreError: 83361:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff983348fab000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5917.800869] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 01:51:46 (1788587506)
[ 5939.247105] LustreError: 68283:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5939.265664] LustreError: 68283:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5943.255178] LustreError: 68283:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5943.268017] LustreError: 68283:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5952.498720] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 01:52:22 (1788587542)
[ 6004.593510] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 01:53:14 (1788587594)
[ 6070.352990] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 01:54:19 (1788587659)
[ 6107.339928] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 01:54:56 (1788587696)
[ 6108.088225] Lustre: Mounted lustre-client - version 2.17.57_104_g4a8f637
[ 6231.768168] Lustre: Unmounted lustre-client
[ 6240.860834] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 01:57:10 (1788587830)
[ 6241.873660] LustreError: 87693:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 6245.235564] Lustre: lustre-OST0000-osc-ffff983348fab000: Connection to lustre-OST0000 (at 192.168.201.111@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6245.247831] Lustre: Skipped 14 previous similar messages
[ 6245.271508] LustreError: lustre-OST0000-osc-ffff983348fab000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6245.288799] LustreError: 87775:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff983348fab000: namespace resource [0x240000400:0x380d:0x0].0x0 (ffff983346bd9400) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6245.323311] LustreError: 87775:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 6245.672613] LustreError: 87693:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout interrupted
[ 6245.681806] Lustre: 2363:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.111@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 6254.966957] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 01:57:24 (1788587844)
[ 6312.929776] Lustre: 88361:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788587849/real 1788587849]  req@ffff9833441d0a80 x1875463819966720/t0(0) o101->lustre-MDT0000-mdc-ffff983348fab000@192.168.201.111@tcp:12/10 lens 328/344 e 0 to 1 dl 1788587904 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 6312.974671] Lustre: 88361:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
[ 6313.012151] Lustre: lustre-MDT0000-mdc-ffff983348fab000: Connection restored to 192.168.201.111@tcp (at 192.168.201.111@tcp)
[ 6313.023414] Lustre: Skipped 18 previous similar messages
[ 6321.433200] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 01:58:30 (1788587910)
[ 6323.384639] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 6326.063706] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 01:58:35 (1788587915)
[ 6391.919758] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6393.978869] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 01:59:43 (1788587983)
[ 6453.069370] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 02:00:42 (1788588042)
[ 6455.785569] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6458.087136] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 02:00:47 (1788588047)
[ 6459.498428] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6461.364891] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 02:00:51 (1788588051)
[ 6505.986056] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 02:01:35 (1788588095)
[ 6522.494433] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6553.568382] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 192.168.201.111@tcp) was lost; in progress operations using this service will fail
[ 6553.592799] LustreError: Skipped 2 previous similar messages
[ 6563.955097] Lustre: Evicted from MGS (at 192.168.201.111@tcp) after server handle changed from 0xc627f82fcc4bf8dd to 0xc627f82fcc4c11d4
[ 6563.963984] Lustre: Skipped 2 previous similar messages
[ 6585.707967] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6586.922627] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6601.457781] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 02:03:11 (1788588191)
[ 6604.438508] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6606.763735] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 02:03:16 (1788588196)
[ 6626.784285] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 192.168.201.111@tcp) was lost; in progress operations using this service will fail
[ 6626.808743] Lustre: Evicted from MGS (at 192.168.201.111@tcp) after server handle changed from 0xc627f82fcc4c11d4 to 0xc627f82fcc4c1593
[ 6641.765157] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 02:03:51 (1788588231)
[ 6711.255588] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 02:05:00 (1788588300)
[ 6721.350237] LustreError: lustre-OST0000-osc-ffff983348fab000: operation ost_setattr to node 192.168.201.111@tcp failed: rc = -107
[ 6721.353316] LustreError: Skipped 9 previous similar messages
[ 6788.277518] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6793.344982] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6861.810991] Lustre: lustre-MDT0000-mdc-ffff983348fab000: Connection to lustre-MDT0000 (at 192.168.201.111@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6861.839883] Lustre: Skipped 7 previous similar messages
[ 6883.295384] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 192.168.201.111@tcp) was lost; in progress operations using this service will fail
[ 6883.328955] LustreError: Skipped 1 previous similar message
[ 6893.568497] Lustre: Evicted from MGS (at 192.168.201.111@tcp) after server handle changed from 0xc627f82fcc4c191a to 0xc627f82fcc4c2767
[ 6893.574975] Lustre: Skipped 1 previous similar message
[ 6907.430194] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6909.021940] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6933.471344] Lustre: 2365:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788588508/real 1788588508]  req@ffff9833664fbb80 x1875463826517376/t0(0) o400->MGC192.168.201.111@tcp@192.168.201.111@tcp:26/25 lens 224/224 e 0 to 1 dl 1788588524 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6933.543749] Lustre: 2365:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 15 previous similar messages
[ 6943.734234] Lustre: MGC192.168.201.111@tcp: Connection restored to 192.168.201.111@tcp (at 192.168.201.111@tcp)
[ 6943.753940] Lustre: Skipped 10 previous similar messages
[ 6962.082606] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6964.521949] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 7104.782549] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 02:11:34 (1788588694)
[ 7175.532451] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 7180.698301] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 7436.701139] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 02:17:06 (1788589026)
[ 7593.452505] Lustre: lustre-MDT0000-mdc-ffff983348fab000: Connection to lustre-MDT0000 (at 192.168.201.111@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7593.473243] Lustre: Skipped 2 previous similar messages
[ 7593.480492] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 192.168.201.111@tcp) was lost; in progress operations using this service will fail
[ 7593.494172] LustreError: Skipped 1 previous similar message
[ 7593.525498] Lustre: Evicted from MGS (at 192.168.201.111@tcp) after server handle changed from 0xc627f82fcc4c2a15 to 0xc627f82fcc5bcd20
[ 7593.541091] Lustre: Skipped 1 previous similar message
[ 7593.557758] Lustre: MGC192.168.201.111@tcp: Connection restored to 192.168.201.111@tcp (at 192.168.201.111@tcp)
[ 7593.578245] Lustre: Skipped 1 previous similar message
[ 7623.136096] Lustre: 2365:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788589174/real 1788589174]  req@ffff983366564700 x1875463837673984/t0(0) o400->lustre-MDT0000-mdc-ffff983348fab000@192.168.201.111@tcp:12/10 lens 224/224 e 0 to 1 dl 1788589214 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7623.163367] Lustre: 2365:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
[ 7627.252363] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 02:20:17 (1788589217)
[ 7628.729432] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7630.796425] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 02:20:20 (1788589220)
[ 7632.423569] LustreError: lustre-MDT0000-mdc-ffff983348fab000: operation ldlm_enqueue to node 192.168.201.111@tcp failed: rc = -107
[ 7632.434135] LustreError: Skipped 149 previous similar messages
[ 7632.460119] LustreError: lustre-MDT0000-mdc-ffff983348fab000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7632.481354] LustreError: 102067:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff983348fab000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff98336b7ba000) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 7632.501406] LustreError: 102067:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 7641.735975] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 02:20:30 (1788589230)
[ 7809.716564] LustreError: lustre-OST0000-osc-ffff983348fab000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7811.510573] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 02:23:21 (1788589401)
[ 7856.544781] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 02:24:06 (1788589446)
[ 7858.546653] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7860.253760] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 02:24:10 (1788589450)
[ 7861.750086] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7863.491879] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 02:24:13 (1788589453)
[ 7897.636619] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 02:24:47 (1788589487)
[ 7933.440930] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 192.168.201.111@tcp) was lost; in progress operations using this service will fail
[ 7933.487668] Lustre: Evicted from MGS (at 192.168.201.111@tcp) after server handle changed from 0xc627f82fcc5bcd20 to 0xc627f82fcc5bf65c
[ 7933.502304] Lustre: Skipped 1 previous similar message
[ 7951.908394] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 02:25:41 (1788589541)
[ 7953.604968] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7956.036970] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 02:25:45 (1788589545)
[ 7957.719653] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7959.298622] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 02:25:49 (1788589549)
[ 7971.365537] Lustre: Unmounted lustre-client
[ 7973.485816] Lustre: Mounted lustre-client - version 2.17.57_104_g4a8f637
[ 7976.480357] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 8015.657888] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2