[ 1829.145256] LustreError: lustre-OST0000-osc-ffff91a190ac0800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1829.169428] Lustre: 2353:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.206.124@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1837.306461] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 11:34:18 (1786721658)
[ 1839.118763] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1840.969636] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 11:34:22 (1786721662)
[ 1842.840818] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1844.590489] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 11:34:25 (1786721665)
[ 1848.362570] LustreError: lustre-MDT0000-mdc-ffff91a190ac0800: operation mds_reint to node 192.168.206.124@tcp failed: rc = -19
[ 1848.371889] LustreError: Skipped 1 previous similar message
[ 1866.719254] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 192.168.206.124@tcp) was lost; in progress operations using this service will fail
[ 1876.908794] Lustre: Evicted from MGS (at 192.168.206.124@tcp) after server handle changed from 0x46d0fea6a0cd51c1 to 0x46d0fea6a0cd6b3d
[ 1969.326599] LustreError: lustre-MDT0000-mdc-ffff91a190ac0800: operation ldlm_enqueue to node 192.168.206.124@tcp failed: rc = -19
[ 1969.358552] LustreError: Skipped 1 previous similar message
[ 1989.599232] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 192.168.206.124@tcp) was lost; in progress operations using this service will fail
[ 1989.620330] Lustre: Evicted from MGS (at 192.168.206.124@tcp) after server handle changed from 0x46d0fea6a0cd6b3d to 0x46d0fea6a0ceefe7
[ 1990.891160] Lustre: 15922:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.206.124@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2014.283975] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 11:37:15 (1786721835)
[ 2014.758307] Lustre: *** cfs_fail_loc=305, val=0***
[ 2014.760096] Lustre: Skipped 2 previous similar messages
[ 2034.542254] LustreError: lustre-OST0001-osc-ffff91a190ac0800: operation ost_connect to node 192.168.206.124@tcp failed: rc = -75
[ 2061.279912] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 192.168.206.124@tcp) was lost; in progress operations using this service will fail
[ 2061.314860] Lustre: Evicted from MGS (at 192.168.206.124@tcp) after server handle changed from 0x46d0fea6a0ceefe7 to 0x46d0fea6a0cef558
[ 2062.844973] Lustre: 15922:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.206.124@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2082.747243] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2084.111638] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2092.252911] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 11:38:33 (1786721913)
[ 2106.350502] Lustre: lustre-MDT0000-mdc-ffff91a190ac0800: Connection to lustre-MDT0000 (at 192.168.206.124@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2106.364394] Lustre: Skipped 12 previous similar messages
[ 2106.368724] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 192.168.206.124@tcp) was lost; in progress operations using this service will fail
[ 2106.392133] Lustre: Evicted from MGS (at 192.168.206.124@tcp) after server handle changed from 0x46d0fea6a0cef558 to 0x46d0fea6a0cefb4e
[ 2106.407789] Lustre: MGC192.168.206.124@tcp: Connection restored to 192.168.206.124@tcp (at 192.168.206.124@tcp)
[ 2106.418689] Lustre: Skipped 15 previous similar messages
[ 2106.457352] LustreError: lustre-MDT0000-mdc-ffff91a190ac0800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2125.867535] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 11:39:06 (1786721946)
[ 2139.615498] LustreError: lustre-OST0000-osc-ffff91a190ac0800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2164.366097] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 11:39:44 (1786721984)
[ 2178.177925] LustreError: lustre-MDT0000-mdc-ffff91a190ac0800: operation ldlm_enqueue to node 192.168.206.124@tcp failed: rc = -19
[ 2199.520036] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 192.168.206.124@tcp) was lost; in progress operations using this service will fail
[ 2199.551330] Lustre: Evicted from MGS (at 192.168.206.124@tcp) after server handle changed from 0x46d0fea6a0cefb4e to 0x46d0fea6a0cf3f41
[ 2200.889443] Lustre: 15922:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.206.124@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2225.930113] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2228.137858] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2311.647219] Lustre: 2354:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786722117/real 1786722117]  req@ffff91a185abad80 x1873511598722944/t0(0) o400->MGC192.168.206.124@tcp@192.168.206.124@tcp:26/25 lens 224/224 e 0 to 1 dl 1786722133 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2311.694615] Lustre: 2354:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 14 previous similar messages
[ 2311.705370] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 192.168.206.124@tcp) was lost; in progress operations using this service will fail
[ 2321.960594] Lustre: Evicted from MGS (at 192.168.206.124@tcp) after server handle changed from 0x46d0fea6a0cf3f41 to 0x46d0fea6a0d0d58e
[ 2336.938829] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2338.703250] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2422.751315] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 192.168.206.124@tcp) was lost; in progress operations using this service will fail
[ 2432.948259] Lustre: Evicted from MGS (at 192.168.206.124@tcp) after server handle changed from 0x46d0fea6a0d0d58e to 0x46d0fea6a0d28dd1
[ 2448.930449] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2452.419776] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2481.985173] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 11:45:03 (1786722303)
[ 2486.089378] LustreError: lustre-MDT0000-mdc-ffff91a190ac0800: operation mds_close to node 192.168.206.124@tcp failed: rc = -19
[ 2486.104669] LustreError: Skipped 3 previous similar messages
[ 2508.781898] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 192.168.206.124@tcp) was lost; in progress operations using this service will fail
[ 2508.807343] Lustre: Evicted from MGS (at 192.168.206.124@tcp) after server handle changed from 0x46d0fea6a0d28dd1 to 0x46d0fea6a0d33d41
[ 2514.896402] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2541.063927] Lustre: 15922:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.206.124@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2559.534848] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2593.155614] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2633.239690] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2674.657954] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 192.168.206.124@tcp) was lost; in progress operations using this service will fail
[ 2674.673708] LustreError: Skipped 3 previous similar messages
[ 2684.912922] Lustre: Evicted from MGS (at 192.168.206.124@tcp) after server handle changed from 0x46d0fea6a0d3a6d4 to 0x46d0fea6a0d3faa3
[ 2684.921621] Lustre: Skipped 3 previous similar messages
[ 2687.271555] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2715.631948] Lustre: lustre-MDT0000-mdc-ffff91a190ac0800: Connection to lustre-MDT0000 (at 192.168.206.124@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2715.662448] Lustre: Skipped 9 previous similar messages
[ 2734.073330] Lustre: MGC192.168.206.124@tcp: Connection restored to 192.168.206.124@tcp (at 192.168.206.124@tcp)
[ 2734.078748] Lustre: Skipped 18 previous similar messages
[ 2743.747768] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2831.307981] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 11:50:52 (1786722652)
[ 2882.619921] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 2884.717074] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3178.897702] LustreError: lustre-OST0000-osc-ffff91a190ac0800: operation ldlm_enqueue to node 192.168.206.124@tcp failed: rc = -19
[ 3178.913623] LustreError: Skipped 10 previous similar messages
[ 3229.873858] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3232.737194] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3513.005245] Lustre: lustre-OST0000-osc-ffff91a190ac0800: Connection to lustre-OST0000 (at 192.168.206.124@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3513.023429] Lustre: Skipped 3 previous similar messages
[ 3537.617036] Lustre: lustre-OST0000-osc-ffff91a190ac0800: Connection restored to 192.168.206.124@tcp (at 192.168.206.124@tcp)
[ 3537.630681] Lustre: Skipped 5 previous similar messages
[ 3560.680757] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3563.819084] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3807.800825] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 12:07:08 (1786723628)
[ 3825.631165] Lustre: 50558:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786723631/real 1786723631]  req@ffff91a185b1e300 x1873511618253440/t0(0) o101->lustre-MDT0000-mdc-ffff91a190ac0800@192.168.206.124@tcp:12/10 lens 576/1152 e 0 to 1 dl 1786723647 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3825.651668] Lustre: 50558:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
[ 3832.661670] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 12:07:33 (1786723653)
[ 3857.831491] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 12:07:58 (1786723678)
[ 3883.792175] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 12:08:24 (1786723704)
[ 3884.430601] Lustre: Mounted lustre-client - version 2.17.57_1_g689e325
[ 3915.999119] Lustre: 2356:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786723722/real 1786723722]  req@ffff91a186c04000 x1873511618272128/t0(0) o400->lustre-MDT0000-mdc-ffff91a190ac0800@192.168.206.124@tcp:12/10 lens 224/224 e 0 to 1 dl 1786723738 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3916.054714] Lustre: 2356:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 3916.255382] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 192.168.206.124@tcp) was lost; in progress operations using this service will fail
[ 3916.270564] LustreError: Skipped 2 previous similar messages
[ 3926.529978] Lustre: Evicted from MGS (at 192.168.206.124@tcp) after server handle changed from 0x46d0fea6a0d56251 to 0x46d0fea6a0e79dd0
[ 3926.539866] Lustre: Skipped 2 previous similar messages
[ 3943.445997] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 3945.013218] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3947.042729] Lustre: Unmounted lustre-client
[ 3953.053327] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 12:09:34 (1786723774)
[ 4072.927150] Lustre: 2353:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786723879/real 1786723879]  req@ffff91a185b1ed80 x1873511618306816/t0(0) o4->lustre-OST0000-osc-ffff91a190ac0800@192.168.206.124@tcp:6/4 lens 488/448 e 0 to 1 dl 1786723895 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4072.956935] Lustre: 2353:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 70 previous similar messages
[ 4121.059742] Lustre: lustre-OST0000-osc-ffff91a190ac0800: Connection to lustre-OST0000 (at 192.168.206.124@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4121.091309] Lustre: Skipped 14 previous similar messages
[ 4152.814058] Lustre: lustre-OST0000-osc-ffff91a190ac0800: Connection restored to 192.168.206.124@tcp (at 192.168.206.124@tcp)
[ 4152.823523] Lustre: Skipped 17 previous similar messages
[ 4248.682255] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 12:14:29 (1786724069)
[ 4300.409928] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 12:15:20 (1786724120)
[ 4305.387387] Lustre: Unmounted lustre-client
[ 4333.148364] Lustre: Mounted lustre-client - version 2.17.57_1_g689e325
[ 4340.890379] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 12:16:01 (1786724161)
[ 4341.330515] LustreError: 56483:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4342.383948] LustreError: 56483:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4342.572399] Lustre: *** cfs_fail_loc=305, val=0***
[ 4367.975490] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 12:16:28 (1786724188)
[ 4368.383634] Lustre: Mounted lustre-client - version 2.17.57_1_g689e325
[ 4371.290691] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4381.602321] Lustre: Unmounted lustre-client
[ 4389.499636] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 12:16:50 (1786724210)
[ 4478.739358] LustreError: lustre-MDT0000-mdc-ffff91a1881d3000: operation mds_reint to node 192.168.206.124@tcp failed: rc = -19
[ 4478.749274] LustreError: Skipped 1 previous similar message
[ 4497.383265] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 192.168.206.124@tcp) was lost; in progress operations using this service will fail
[ 4497.402698] Lustre: Evicted from MGS (at 192.168.206.124@tcp) after server handle changed from 0x46d0fea6a0e7b86b to 0x46d0fea6a0ea7218
[ 4627.101228] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 12:20:48 (1786724448)
[ 4631.266676] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4645.866678] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 192.168.206.124@tcp) was lost; in progress operations using this service will fail
[ 4645.894079] Lustre: Evicted from MGS (at 192.168.206.124@tcp) after server handle changed from 0x46d0fea6a0ea7218 to 0x46d0fea6a0ef34f1
[ 4645.945210] LustreError: lustre-MDT0000-mdc-ffff91a1881d3000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4658.543829] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 12:21:19 (1786724479)
[ 4658.988564] Lustre: Mounted lustre-client - version 2.17.57_1_g689e325
[ 4666.195185] LustreError: lustre-OST0000-osc-ffff91a1881d3000: operation ldlm_enqueue to node 192.168.206.124@tcp failed: rc = -107
[ 4666.221951] LustreError: lustre-OST0000-osc-ffff91a1881d3000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4676.575185] Lustre: 2355:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786724482/real 1786724482]  req@ffff91a18b74d180 x1873511622959872/t0(0) o103->lustre-OST0000-osc-ffff91a190084000@192.168.206.124@tcp:17/18 lens 328/224 e 0 to 1 dl 1786724498 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295
[ 4676.629070] Lustre: 2355:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 89 previous similar messages
[ 4679.450161] Lustre: Unmounted lustre-client
[ 4686.812898] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 12:21:47 (1786724507)
[ 4693.430475] LustreError: lustre-MDT0000-mdc-ffff91a1881d3000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4693.438121] LustreError: 60864:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4693.439808] LustreError: 60948:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff91a1881d3000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff91a18665a200) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4700.639221] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 12:22:01 (1786724521)
[ 4700.892744] LustreError: 61568:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4706.130661] LustreError: lustre-MDT0000-mdc-ffff91a1881d3000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4706.149912] LustreError: 61586:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4706.158691] LustreError: 61568:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=1
[ 4706.169135] LustreError: 61568:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting
[ 4706.214938] LustreError: 61591:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4707.269253] LustreError: 61597:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4707.275936] LustreError: 61597:0:(file.c:6156:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4709.500854] LustreError: 61619:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4709.513763] LustreError: 61619:0:(file.c:6156:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4713.938358] LustreError: 61664:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4713.950444] LustreError: 61664:0:(file.c:6156:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4723.999974] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 12:22:24 (1786724544)
[ 4731.889704] Lustre: lustre-OST0000-osc-ffff91a1881d3000: Connection to lustre-OST0000 (at 192.168.206.124@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4731.908739] Lustre: Skipped 18 previous similar messages
[ 4762.649902] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4764.122915] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4777.619682] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 12:23:18 (1786724598)
[ 4802.303720] Lustre: lustre-OST0000-osc-ffff91a1881d3000: Connection restored to 192.168.206.124@tcp (at 192.168.206.124@tcp)
[ 4802.321911] Lustre: Skipped 17 previous similar messages
[ 4817.816893] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4819.694266] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4830.890959] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 12:24:11 (1786724651)
[ 4899.102597] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4901.711280] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4911.492046] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 12:25:31 (1786724731)
[ 4949.015113] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4950.408861] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4956.482040] Lustre: Unmounted lustre-client
[ 5010.615404] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5013.150823] Lustre: Mounted lustre-client - version 2.17.57_1_g689e325
[ 5019.419123] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 12:27:20 (1786724840)
[ 5021.794854] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5023.309715] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 12:27:24 (1786724844)
[ 5051.461082] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 12:27:52 (1786724872)
[ 5052.566463] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5054.100435] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 12:27:55 (1786724875)
[ 5055.211333] Lustre: *** cfs_fail_loc=805, val=0***
[ 5055.238853] Lustre: Mounted lustre-client - version 2.17.57_1_g689e325
[ 5059.564280] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5081.081691] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 192.168.206.124@tcp) was lost; in progress operations using this service will fail
[ 5081.101257] Lustre: Evicted from MGS (at 192.168.206.124@tcp) after server handle changed from 0x46d0fea6a0ef3ed7 to 0x46d0fea6a0ef40f2
[ 5082.229032] Lustre: 68250:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.206.124@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 5098.580816] Lustre: Unmounted lustre-client
[ 5107.130377] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 12:28:47 (1786724927)
[ 5149.526367] LustreError: 2352:0:(client.c:3435:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff91a18b690000 x1873511623046016/t94489280517(94489280517) o101->lustre-MDT0000-mdc-ffff91a190954800@192.168.206.124@tcp:12/10 lens 648/608 e 0 to 0 dl 1786724987 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5155.098262] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5157.213766] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5164.942516] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 12:29:45 (1786724985)
[ 5166.414515] LustreError: lustre-OST0000-osc-ffff91a190954800: operation ost_write to node 192.168.206.124@tcp failed: rc = -107
[ 5166.428831] LustreError: Skipped 2 previous similar messages
[ 5166.442565] LustreError: lustre-OST0000-osc-ffff91a190954800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5166.456914] Lustre: 2354:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.206.124@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5166.497767] LustreError: 72789:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff91a190954800: namespace resource [0x240000400:0x38c2:0x0].0x0 (ffff91a186436b00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5174.662317] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 12:29:55 (1786724995)
[ 5176.297269] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5178.450524] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 12:29:59 (1786724999)
[ 5180.755745] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5183.299605] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 12:30:03 (1786725003)
[ 5185.043655] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5187.327425] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 12:30:08 (1786725008)
[ 5189.676851] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5192.199337] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 12:30:12 (1786725012)
[ 5193.789404] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5196.366507] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 12:30:16 (1786725016)
[ 5198.043083] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5200.106764] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 12:30:20 (1786725020)
[ 5201.937780] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5204.286206] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 12:30:24 (1786725024)
[ 5206.023108] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5207.668962] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 12:30:28 (1786725028)
[ 5208.969552] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5210.706200] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 12:30:31 (1786725031)
[ 5212.178155] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5213.885391] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 12:30:34 (1786725034)
[ 5215.466550] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5217.198522] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 12:30:38 (1786725038)
[ 5219.327199] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5220.852274] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 12:30:42 (1786725042)
[ 5251.017545] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 12:31:12 (1786725072)
[ 5282.734603] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 12:31:43 (1786725103)
[ 5283.345674] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5297.382695] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 12:31:57 (1786725117)
[ 5297.765062] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5309.193391] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 12:32:10 (1786725130)
[ 5309.543652] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5318.075792] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 12:32:19 (1786725139)
[ 5318.466209] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5327.028413] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 12:32:27 (1786725147)
[ 5327.418647] Lustre: *** cfs_fail_loc=510, val=3***
[ 5335.510538] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 12:32:36 (1786725156)
[ 5336.011298] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5345.771570] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 12:32:46 (1786725166)
[ 5346.217718] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5355.487151] Lustre: 2356:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786725161/real 1786725161]  req@ffff91a18b74c700 x1873511623087488/t0(0) o400->lustre-OST0000-osc-ffff91a190954800@192.168.206.124@tcp:28/4 lens 224/224 e 0 to 1 dl 1786725177 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5355.518062] Lustre: 2356:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 18 previous similar messages
[ 5405.663423] Lustre: lustre-OST0000-osc-ffff91a190954800: Connection to lustre-OST0000 (at 192.168.206.124@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5405.674579] Lustre: Skipped 9 previous similar messages
[ 5413.243104] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 12:33:53 (1786725233)
[ 5413.561217] LustreError: 82995:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5416.513174] LustreError: lustre-MDT0000-mdc-ffff91a190954800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5416.549351] LustreError: 83009:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff91a190954800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff91a186436100) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5416.598818] Lustre: lustre-MDT0000-mdc-ffff91a190954800: Connection restored to 192.168.206.124@tcp (at 192.168.206.124@tcp)
[ 5416.621366] Lustre: Skipped 10 previous similar messages
[ 5417.631115] LustreError: 82995:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5419.716292] LustreError: 83025:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5422.690030] LustreError: lustre-MDT0000-mdc-ffff91a190954800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5422.715422] LustreError: 83039:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff91a190954800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff91a186436800) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5422.750472] LustreError: 83039:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5423.807117] LustreError: 83025:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5430.962760] LustreError: lustre-MDT0000-mdc-ffff91a190954800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5431.551704] LustreError: 83080:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5434.554887] LustreError: lustre-MDT0000-mdc-ffff91a190954800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5434.631123] LustreError: 83100:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5434.638346] LustreError: 83100:0:(file.c:6156:ll_inode_revalidate_fini()) Skipped 20 previous similar messages
[ 5435.631987] LustreError: 83080:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5435.639900] LustreError: 83080:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff91a190954800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5435.755671] LustreError: 83106:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5435.764304] LustreError: 83106:0:(file.c:6156:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5438.014078] LustreError: 83128:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5438.032407] LustreError: 83128:0:(file.c:6156:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5441.681635] LustreError: 83155:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5444.539632] LustreError: lustre-MDT0000-mdc-ffff91a190954800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5444.567598] LustreError: 83169:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff91a190954800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff91a1866acd00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5444.581055] LustreError: 83169:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5445.767137] LustreError: 83155:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5452.806923] LustreError: lustre-MDT0000-mdc-ffff91a190954800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5453.145549] LustreError: 83211:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5456.062095] LustreError: lustre-MDT0000-mdc-ffff91a190954800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5456.088554] LustreError: 83225:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff91a190954800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff91a186436600) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5456.131443] LustreError: 83225:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5457.225746] LustreError: 83211:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5460.155512] LustreError: lustre-MDT0000-mdc-ffff91a190954800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5460.220797] LustreError: 83258:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5460.235702] LustreError: 83258:0:(file.c:6156:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5461.445497] LustreError: 83239:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff91a190954800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5469.324086] LustreError: 83310:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5469.333267] LustreError: 83310:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5472.843293] LustreError: lustre-MDT0000-mdc-ffff91a190954800: operation ldlm_enqueue to node 192.168.206.124@tcp failed: rc = -107
[ 5472.852500] LustreError: Skipped 8 previous similar messages
[ 5472.871422] LustreError: lustre-MDT0000-mdc-ffff91a190954800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5472.974399] LustreError: 83332:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5472.979532] LustreError: 83332:0:(file.c:6156:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5473.407130] LustreError: 83310:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5473.419919] LustreError: 83310:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5473.444619] LustreError: 83310:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff91a190954800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5483.239633] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 12:35:04 (1786725304)
[ 5502.924337] LustreError: 68245:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5502.933304] LustreError: 68245:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5507.007932] LustreError: 68245:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5507.012014] LustreError: 68245:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5512.700822] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 12:35:33 (1786725333)
[ 5560.870192] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 12:36:21 (1786725381)
[ 5625.655256] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 12:37:26 (1786725446)
[ 5657.834240] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 12:37:59 (1786725479)
[ 5658.359802] Lustre: Mounted lustre-client - version 2.17.57_1_g689e325
[ 5781.414435] Lustre: Unmounted lustre-client
[ 5787.922303] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 12:40:09 (1786725609)
[ 5788.325331] LustreError: 87636:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5790.498147] LustreError: lustre-OST0000-osc-ffff91a190954800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5790.509523] LustreError: 87723:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff91a190954800: namespace resource [0x240000400:0x38ed:0x0].0x0 (ffff91a186436e00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5790.524119] LustreError: 87723:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5790.743153] LustreError: 87636:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5790.753362] Lustre: 2355:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.206.124@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5795.824576] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 12:40:17 (1786725617)
[ 5860.573605] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 12:41:21 (1786725681)
[ 5862.123648] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 5864.100550] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 12:41:24 (1786725684)
[ 5926.780941] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 5928.201781] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 12:42:29 (1786725749)
[ 5985.247160] Lustre: 90035:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786725752/real 1786725752]  req@ffff91a18b74f800 x1873511623230080/t0(0) o36->lustre-MDT0000-mdc-ffff91a190954800@192.168.206.124@tcp:12/10 lens 496/440 e 0 to 1 dl 1786725807 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ln.0' uid:0 gid:0 projid:4294967295
[ 5985.275107] Lustre: 90035:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
[ 5986.502704] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 12:43:27 (1786725807)
[ 5988.319959] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 5989.746536] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 12:43:30 (1786725810)
[ 5991.259951] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 5992.768733] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 12:43:33 (1786725813)
[ 6021.638990] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 12:44:02 (1786725842)
[ 6031.326312] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6056.928675] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 192.168.206.124@tcp) was lost; in progress operations using this service will fail
[ 6056.934672] LustreError: Skipped 2 previous similar messages
[ 6056.943827] Lustre: Evicted from MGS (at 192.168.206.124@tcp) after server handle changed from 0x46d0fea6a0ef4d32 to 0x46d0fea6a0ef655e
[ 6056.949169] Lustre: Skipped 2 previous similar messages
[ 6056.953473] Lustre: MGC192.168.206.124@tcp: Connection restored to 192.168.206.124@tcp (at 192.168.206.124@tcp)
[ 6056.956969] Lustre: Skipped 15 previous similar messages
[ 6058.674527] Lustre: 68250:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.206.124@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6063.071276] Lustre: lustre-MDT0000-mdc-ffff91a190954800: Connection to lustre-MDT0000 (at 192.168.206.124@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6063.102628] Lustre: Skipped 16 previous similar messages
[ 6082.497631] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6083.879408] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6094.080751] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 12:45:14 (1786725914)
[ 6096.988486] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6099.018260] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 12:45:19 (1786725919)
[ 6114.295988] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 192.168.206.124@tcp) was lost; in progress operations using this service will fail
[ 6114.321444] Lustre: Evicted from MGS (at 192.168.206.124@tcp) after server handle changed from 0x46d0fea6a0ef655e to 0x46d0fea6a0ef6a19
[ 6123.546454] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 12:45:44 (1786725944)
[ 6156.670908] Lustre: 68250:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.206.124@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6178.189626] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 12:46:38 (1786725998)
[ 6185.677558] LustreError: lustre-OST0000-osc-ffff91a190954800: operation ost_setattr to node 192.168.206.124@tcp failed: rc = -107
[ 6185.683637] LustreError: Skipped 1 previous similar message
[ 6236.060672] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6239.965220] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6329.119764] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 192.168.206.124@tcp) was lost; in progress operations using this service will fail
[ 6329.165702] LustreError: Skipped 1 previous similar message
[ 6329.191341] Lustre: Evicted from MGS (at 192.168.206.124@tcp) after server handle changed from 0x46d0fea6a0ef6c26 to 0x46d0fea6a0ef7a7a
[ 6329.207868] Lustre: Skipped 1 previous similar message
[ 6330.548033] Lustre: 68250:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.206.124@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6354.489927] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6356.221992] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6394.167056] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6396.050960] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6529.612223] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 12:52:30 (1786726350)
[ 6589.605512] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6592.492729] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6850.977813] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 12:57:51 (1786726671)
[ 6973.416831] Lustre: lustre-MDT0000-mdc-ffff91a190954800: Connection to lustre-MDT0000 (at 192.168.206.124@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6973.433613] Lustre: Skipped 6 previous similar messages
[ 6988.772849] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 192.168.206.124@tcp) was lost; in progress operations using this service will fail
[ 6988.780388] LustreError: Skipped 1 previous similar message
[ 6988.791046] Lustre: Evicted from MGS (at 192.168.206.124@tcp) after server handle changed from 0x46d0fea6a0ef7f82 to 0x46d0fea6a0ff227f
[ 6988.797138] Lustre: Skipped 1 previous similar message
[ 6988.802347] Lustre: MGC192.168.206.124@tcp: Connection restored to 192.168.206.124@tcp (at 192.168.206.124@tcp)
[ 6988.807621] Lustre: Skipped 9 previous similar messages
[ 7020.631861] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 13:00:41 (1786726841)
[ 7021.884452] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7023.238364] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 13:00:44 (1786726844)
[ 7025.007392] LustreError: lustre-MDT0000-mdc-ffff91a190954800: operation ldlm_enqueue to node 192.168.206.124@tcp failed: rc = -107
[ 7025.017448] LustreError: Skipped 296 previous similar messages
[ 7025.026947] LustreError: lustre-MDT0000-mdc-ffff91a190954800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7031.987132] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 13:00:53 (1786726853)
[ 7199.727220] LustreError: lustre-OST0000-osc-ffff91a190954800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7201.546817] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 13:03:42 (1786727022)
[ 7226.015575] Lustre: 2355:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786727028/real 1786727028]  req@ffff91a18982f480 x1873511640947840/t0(0) o4->lustre-OST0000-osc-ffff91a190954800@192.168.206.124@tcp:6/4 lens 4584/448 e 0 to 1 dl 1786727048 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7226.036392] Lustre: 2355:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 17 previous similar messages
[ 7243.646048] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 13:04:24 (1786727064)
[ 7245.272628] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7247.299823] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 13:04:28 (1786727068)
[ 7248.563247] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7250.374382] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 13:04:31 (1786727071)
[ 7283.589461] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 13:05:04 (1786727104)
[ 7323.642684] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 192.168.206.124@tcp) was lost; in progress operations using this service will fail
[ 7323.676895] Lustre: Evicted from MGS (at 192.168.206.124@tcp) after server handle changed from 0x46d0fea6a0ff227f to 0x46d0fea6a0ff4906
[ 7323.688561] Lustre: Skipped 1 previous similar message
[ 7338.472820] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 13:05:59 (1786727159)
[ 7339.925730] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7341.469216] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 13:06:02 (1786727162)
[ 7343.062498] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7344.460355] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 13:06:05 (1786727165)
[ 7356.254187] Lustre: Unmounted lustre-client
[ 7357.862600] Lustre: Mounted lustre-client - version 2.17.57_1_g689e325
[ 7360.449873] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7379.423420] Lustre: 2356:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786727185/real 1786727185]  req@ffff91a18982ea00 x1873511641134080/t0(0) o400->MGC192.168.206.124@tcp@192.168.206.124@tcp:26/25 lens 224/224 e 0 to 1 dl 1786727201 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7389.407222] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2