[ 1902.612188] LustreError: lustre-OST0000-osc-ffff934418530000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1902.640845] Lustre: 2346:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.133@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1912.216611] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 17:47:22 (1786830442)
[ 1915.201991] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1916.983647] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 17:47:27 (1786830447)
[ 1919.025460] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1921.151656] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 17:47:31 (1786830451)
[ 1926.164685] LustreError: lustre-MDT0000-mdc-ffff934418530000: operation mds_close to node 192.168.202.133@tcp failed: rc = -19
[ 1926.180985] LustreError: Skipped 2 previous similar messages
[ 1943.521779] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 192.168.202.133@tcp) was lost; in progress operations using this service will fail
[ 1953.723233] Lustre: Evicted from MGS (at 192.168.202.133@tcp) after server handle changed from 0x20861098814c0058 to 0x20861098814c19f7
[ 1955.446979] Lustre: 15927:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.202.133@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2062.542038] LustreError: lustre-MDT0000-mdc-ffff934418530000: operation mds_reint to node 192.168.202.133@tcp failed: rc = -19
[ 2082.784274] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 192.168.202.133@tcp) was lost; in progress operations using this service will fail
[ 2082.800207] Lustre: Evicted from MGS (at 192.168.202.133@tcp) after server handle changed from 0x20861098814c19f7 to 0x20861098814ddc82
[ 2083.962766] Lustre: 15927:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.202.133@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2108.576294] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 17:50:39 (1786830639)
[ 2109.058963] Lustre: *** cfs_fail_loc=305, val=0***
[ 2109.060712] Lustre: Skipped 4 previous similar messages
[ 2154.979380] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 192.168.202.133@tcp) was lost; in progress operations using this service will fail
[ 2165.233892] Lustre: Evicted from MGS (at 192.168.202.133@tcp) after server handle changed from 0x20861098814ddc82 to 0x20861098814de18a
[ 2184.802432] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2186.589969] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2195.435928] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 17:52:06 (1786830726)
[ 2211.336180] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 192.168.202.133@tcp) was lost; in progress operations using this service will fail
[ 2211.369838] Lustre: Evicted from MGS (at 192.168.202.133@tcp) after server handle changed from 0x20861098814de18a to 0x20861098814de867
[ 2211.393399] LustreError: lustre-MDT0000-mdc-ffff934418530000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2233.944614] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 17:52:44 (1786830764)
[ 2270.743246] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 17:53:21 (1786830801)
[ 2284.773662] LustreError: lustre-MDT0000-mdc-ffff934418530000: operation ldlm_enqueue to node 192.168.202.133@tcp failed: rc = -19
[ 2284.785290] LustreError: Skipped 1 previous similar message
[ 2303.072738] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 192.168.202.133@tcp) was lost; in progress operations using this service will fail
[ 2303.125871] Lustre: Evicted from MGS (at 192.168.202.133@tcp) after server handle changed from 0x20861098814de867 to 0x20861098814e2bea
[ 2330.276992] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2332.725575] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2419.937863] Lustre: 2348:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786830936/real 1786830936]  req@ffff934404af3800 x1873625564618624/t0(0) o400->MGC192.168.202.133@tcp@192.168.202.133@tcp:26/25 lens 224/224 e 0 to 1 dl 1786830952 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2419.963028] Lustre: 2348:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 18 previous similar messages
[ 2419.969419] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 192.168.202.133@tcp) was lost; in progress operations using this service will fail
[ 2419.984570] Lustre: Evicted from MGS (at 192.168.202.133@tcp) after server handle changed from 0x20861098814e2bea to 0x20861098814fec84
[ 2421.297290] Lustre: 15927:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.202.133@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2448.816559] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2451.198177] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2517.373245] Lustre: lustre-MDT0000-mdc-ffff934418530000: Connection to lustre-MDT0000 (at 192.168.202.133@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2517.397418] Lustre: Skipped 8 previous similar messages
[ 2536.928389] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 192.168.202.133@tcp) was lost; in progress operations using this service will fail
[ 2536.945249] Lustre: Evicted from MGS (at 192.168.202.133@tcp) after server handle changed from 0x20861098814fec84 to 0x208610988151c06c
[ 2536.958510] Lustre: MGC192.168.202.133@tcp: Connection restored to 192.168.202.133@tcp (at 192.168.202.133@tcp)
[ 2536.965796] Lustre: Skipped 14 previous similar messages
[ 2538.352652] Lustre: 15927:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.202.133@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2565.607829] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2567.607623] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2598.676797] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 17:58:49 (1786831129)
[ 2603.228882] LustreError: lustre-MDT0000-mdc-ffff934418530000: operation mds_close to node 192.168.202.133@tcp failed: rc = -19
[ 2603.243256] LustreError: Skipped 4 previous similar messages
[ 2623.712403] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 192.168.202.133@tcp) was lost; in progress operations using this service will fail
[ 2623.745755] Lustre: Evicted from MGS (at 192.168.202.133@tcp) after server handle changed from 0x208610988151c06c to 0x2086109881529c3d
[ 2631.052729] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2663.932810] Lustre: 15927:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.202.133@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2676.055671] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2722.357759] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2752.992588] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 192.168.202.133@tcp) was lost; in progress operations using this service will fail
[ 2753.012173] LustreError: Skipped 2 previous similar messages
[ 2763.177782] Lustre: Evicted from MGS (at 192.168.202.133@tcp) after server handle changed from 0x208610988152fa93 to 0x2086109881535706
[ 2763.183591] Lustre: Skipped 2 previous similar messages
[ 2773.130835] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2834.109326] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2887.319966] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2972.049474] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 18:05:02 (1786831502)
[ 3024.564322] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3026.861902] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3316.229506] LustreError: lustre-OST0000-osc-ffff934418530000: operation ldlm_enqueue to node 192.168.202.133@tcp failed: rc = -107
[ 3316.247624] LustreError: Skipped 8 previous similar messages
[ 3316.254219] Lustre: lustre-OST0000-osc-ffff934418530000: Connection to lustre-OST0000 (at 192.168.202.133@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3316.285855] Lustre: Skipped 7 previous similar messages
[ 3338.243158] Lustre: lustre-OST0000-osc-ffff934418530000: Connection restored to 192.168.202.133@tcp (at 192.168.202.133@tcp)
[ 3338.255518] Lustre: Skipped 14 previous similar messages
[ 3356.551403] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3358.600302] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3691.942357] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3694.079655] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3942.219534] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 18:21:13 (1786832473)
[ 3960.288221] Lustre: 50542:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786832476/real 1786832476]  req@ffff934407eb9880 x1873625589747328/t0(0) o101->lustre-MDT0000-mdc-ffff934418530000@192.168.202.133@tcp:12/10 lens 576/1152 e 0 to 1 dl 1786832492 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3960.321481] Lustre: 50542:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
[ 3960.332700] Lustre: lustre-MDT0000-mdc-ffff934418530000: Connection to lustre-MDT0000 (at 192.168.202.133@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3960.347091] Lustre: Skipped 1 previous similar message
[ 3960.371583] Lustre: lustre-MDT0000-mdc-ffff934418530000: Connection restored to 192.168.202.133@tcp (at 192.168.202.133@tcp)
[ 3960.379638] Lustre: Skipped 1 previous similar message
[ 3968.036637] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 18:21:39 (1786832499)
[ 3993.117254] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 18:22:04 (1786832524)
[ 4017.182756] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 18:22:28 (1786832548)
[ 4017.832321] Lustre: Mounted lustre-client - version 2.17.57_3_g928f38d
[ 4048.869369] Lustre: 2348:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786832565/real 1786832565]  req@ffff934407eb9500 x1873625589765888/t0(0) o400->MGC192.168.202.133@tcp@192.168.202.133@tcp:26/25 lens 224/224 e 0 to 1 dl 1786832581 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4048.935420] Lustre: 2348:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 4048.942642] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 192.168.202.133@tcp) was lost; in progress operations using this service will fail
[ 4048.953068] LustreError: Skipped 3 previous similar messages
[ 4059.131058] Lustre: Evicted from MGS (at 192.168.202.133@tcp) after server handle changed from 0x208610988155582f to 0x20861098816d47e4
[ 4059.141265] Lustre: Skipped 3 previous similar messages
[ 4081.500363] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4083.050836] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4085.552637] Lustre: Unmounted lustre-client
[ 4092.848218] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 18:23:43 (1786832623)
[ 4202.464354] Lustre: 2347:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786832718/real 1786832718]  req@ffff934406a6ea00 x1873625589801088/t0(0) o4->lustre-OST0000-osc-ffff934418530000@192.168.202.133@tcp:6/4 lens 488/448 e 0 to 1 dl 1786832734 ref 2 fl Rpc:XQr/602/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4202.507435] Lustre: 2347:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 53 previous similar messages
[ 4394.555760] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 18:28:45 (1786832925)
[ 4445.368687] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 18:29:36 (1786832976)
[ 4449.295934] Lustre: Unmounted lustre-client
[ 4470.445576] Lustre: Mounted lustre-client - version 2.17.57_3_g928f38d
[ 4476.960575] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 18:30:07 (1786833007)
[ 4477.356591] LustreError: 56470:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4478.400106] LustreError: 56470:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4478.627987] Lustre: *** cfs_fail_loc=305, val=0***
[ 4501.621841] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 18:30:32 (1786833032)
[ 4501.948473] Lustre: Mounted lustre-client - version 2.17.57_3_g928f38d
[ 4504.078595] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4514.408475] Lustre: Unmounted lustre-client
[ 4521.052820] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 18:30:52 (1786833052)
[ 4599.703045] LustreError: lustre-MDT0000-mdc-ffff934406b3b800: operation mds_reint to node 192.168.202.133@tcp failed: rc = -19
[ 4599.715592] LustreError: Skipped 1 previous similar message
[ 4599.722281] Lustre: lustre-MDT0000-mdc-ffff934406b3b800: Connection to lustre-MDT0000 (at 192.168.202.133@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4599.736260] Lustre: Skipped 23 previous similar messages
[ 4618.734319] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 192.168.202.133@tcp) was lost; in progress operations using this service will fail
[ 4618.764292] Lustre: Evicted from MGS (at 192.168.202.133@tcp) after server handle changed from 0x20861098816d6152 to 0x2086109881701a0a
[ 4618.785156] Lustre: MGC192.168.202.133@tcp: Connection restored to 192.168.202.133@tcp (at 192.168.202.133@tcp)
[ 4618.792626] Lustre: Skipped 24 previous similar messages
[ 4753.287068] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 18:34:44 (1786833284)
[ 4758.402272] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4772.350484] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 192.168.202.133@tcp) was lost; in progress operations using this service will fail
[ 4772.372879] Lustre: Evicted from MGS (at 192.168.202.133@tcp) after server handle changed from 0x2086109881701a0a to 0x208610988174de2c
[ 4772.406275] LustreError: lustre-MDT0000-mdc-ffff934406b3b800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4784.706074] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 18:35:15 (1786833315)
[ 4785.003345] Lustre: Mounted lustre-client - version 2.17.57_3_g928f38d
[ 4791.863455] LustreError: lustre-OST0000-osc-ffff934406b3b800: operation ldlm_enqueue to node 192.168.202.133@tcp failed: rc = -107
[ 4791.874875] LustreError: lustre-OST0000-osc-ffff934406b3b800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4802.016286] Lustre: 2346:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786833318/real 1786833318]  req@ffff934404140000 x1873625594454272/t0(0) o103->lustre-OST0000-osc-ffff934418532000@192.168.202.133@tcp:17/18 lens 328/224 e 0 to 1 dl 1786833334 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295
[ 4802.044497] Lustre: 2346:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 95 previous similar messages
[ 4803.783309] Lustre: Unmounted lustre-client
[ 4810.573206] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 18:35:41 (1786833341)
[ 4816.721922] LustreError: lustre-MDT0000-mdc-ffff934406b3b800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4816.732612] LustreError: 60854:0:(file.c:6165:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4822.141745] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 18:35:53 (1786833353)
[ 4822.298986] LustreError: 61556:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4827.292535] LustreError: lustre-MDT0000-mdc-ffff934406b3b800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4827.302950] LustreError: 61574:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4827.311277] LustreError: 61556:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=1
[ 4827.323351] LustreError: 61556:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting
[ 4827.373935] LustreError: 61579:0:(file.c:6165:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4828.454205] LustreError: 61586:0:(file.c:6165:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4828.464482] LustreError: 61586:0:(file.c:6165:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4830.661220] LustreError: 61608:0:(file.c:6165:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4830.672190] LustreError: 61608:0:(file.c:6165:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4835.218883] LustreError: 61652:0:(file.c:6165:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4835.224747] LustreError: 61652:0:(file.c:6165:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4845.184794] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 18:36:16 (1786833376)
[ 4881.849907] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4883.678629] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4893.960149] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 18:37:05 (1786833425)
[ 4932.154309] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4933.958345] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4943.035991] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 18:37:54 (1786833474)
[ 5006.033070] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5007.849663] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5017.130292] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 18:39:07 (1786833547)
[ 5054.856641] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5056.659929] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5061.863147] Lustre: Unmounted lustre-client
[ 5117.795762] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5120.242574] Lustre: Mounted lustre-client - version 2.17.57_3_g928f38d
[ 5127.590683] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 18:40:57 (1786833657)
[ 5130.138097] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5131.969745] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 18:41:02 (1786833662)
[ 5162.184746] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 18:41:32 (1786833692)
[ 5164.408063] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5166.691523] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 18:41:37 (1786833697)
[ 5169.164083] Lustre: *** cfs_fail_loc=805, val=0***
[ 5169.274107] Lustre: Mounted lustre-client - version 2.17.57_3_g928f38d
[ 5176.603729] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5196.260054] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 192.168.202.133@tcp) was lost; in progress operations using this service will fail
[ 5206.505479] Lustre: Evicted from MGS (at 192.168.202.133@tcp) after server handle changed from 0x208610988174e835 to 0x208610988174ebdf
[ 5206.524826] LustreError: 2344:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9344102a6000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff934404177900) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5208.351899] Lustre: Unmounted lustre-client
[ 5215.554722] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 18:42:26 (1786833746)
[ 5232.096245] Lustre: lustre-MDT0000-mdc-ffff934403fd7000: Connection to lustre-MDT0000 (at 192.168.202.133@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5232.107585] Lustre: Skipped 14 previous similar messages
[ 5238.265994] Lustre: MGC192.168.202.133@tcp: Connection restored to 192.168.202.133@tcp (at 192.168.202.133@tcp)
[ 5238.272296] Lustre: Skipped 15 previous similar messages
[ 5239.608880] Lustre: 68224:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.202.133@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 5257.763785] LustreError: 2344:0:(client.c:3435:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff93440a6eb480 x1873625594533888/t90194313220(90194313220) o101->lustre-MDT0000-mdc-ffff934403fd7000@192.168.202.133@tcp:12/10 lens 664/608 e 0 to 0 dl 1786833806 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5262.772188] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5263.876740] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5272.114959] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 18:43:23 (1786833803)
[ 5273.650176] LustreError: lustre-OST0000-osc-ffff934403fd7000: operation ost_write to node 192.168.202.133@tcp failed: rc = -107
[ 5273.656766] LustreError: Skipped 2 previous similar messages
[ 5273.669618] LustreError: lustre-OST0000-osc-ffff934403fd7000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5273.679803] Lustre: 2345:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.133@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5273.712699] LustreError: 72764:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff934403fd7000: namespace resource [0x240000400:0x4462:0x0].0x0 (ffff93440a6f8f00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5273.733709] LustreError: 72764:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5281.206230] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 18:43:32 (1786833812)
[ 5282.433224] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5284.021667] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 18:43:35 (1786833815)
[ 5285.263124] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5286.742750] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 18:43:37 (1786833817)
[ 5287.811930] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5289.500759] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 18:43:40 (1786833820)
[ 5290.602856] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5292.236212] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 18:43:43 (1786833823)
[ 5293.942738] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5296.303161] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 18:43:46 (1786833826)
[ 5297.696077] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5299.285451] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 18:43:50 (1786833830)
[ 5300.388438] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5302.241626] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 18:43:53 (1786833833)
[ 5303.836411] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5305.451889] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 18:43:56 (1786833836)
[ 5306.697328] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5308.440418] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 18:43:59 (1786833839)
[ 5309.836655] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5311.674791] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 18:44:02 (1786833842)
[ 5312.931911] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5314.948842] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 18:44:05 (1786833845)
[ 5317.426348] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5318.826159] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 18:44:09 (1786833849)
[ 5339.632683] Lustre: 68224:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.202.133@tcp: IR log lustre-cliir failed, not fatal: rc = -5
[ 5349.517403] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 18:44:40 (1786833880)
[ 5378.648124] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 18:45:09 (1786833909)
[ 5379.173764] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5389.586583] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 18:45:20 (1786833920)
[ 5389.946614] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5399.363242] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 18:45:30 (1786833930)
[ 5399.908577] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5408.121556] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 18:45:39 (1786833939)
[ 5408.376154] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5419.506226] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 18:45:50 (1786833950)
[ 5419.976729] Lustre: *** cfs_fail_loc=510, val=3***
[ 5430.307431] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 18:46:00 (1786833960)
[ 5430.806516] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5489.760172] Lustre: 2346:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786833967/real 1786833967]  req@ffff934404142d80 x1873625594576384/t0(0) o3->lustre-OST0000-osc-ffff934403fd7000@192.168.202.133@tcp:6/4 lens 488/4536 e 0 to 1 dl 1786834022 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 5489.778692] Lustre: 2346:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 14 previous similar messages
[ 5496.790650] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 18:47:07 (1786834027)
[ 5497.214593] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5563.138712] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 18:48:14 (1786834094)
[ 5563.413261] LustreError: 82980:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5566.409309] LustreError: lustre-MDT0000-mdc-ffff934403fd7000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5566.458477] LustreError: 82994:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff934403fd7000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff934418c43800) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5567.480171] LustreError: 82980:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5569.592389] LustreError: 83010:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5572.787540] LustreError: lustre-MDT0000-mdc-ffff934403fd7000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5572.805914] LustreError: 83024:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff934403fd7000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff93440a6f8100) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5572.827157] LustreError: 83024:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5573.680245] LustreError: 83010:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5581.056922] LustreError: lustre-MDT0000-mdc-ffff934403fd7000: operation ldlm_enqueue to node 192.168.202.133@tcp failed: rc = -107
[ 5581.069054] LustreError: Skipped 2 previous similar messages
[ 5581.090441] LustreError: lustre-MDT0000-mdc-ffff934403fd7000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5581.586365] LustreError: 83066:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5584.610559] LustreError: lustre-MDT0000-mdc-ffff934403fd7000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5584.666614] LustreError: 83086:0:(file.c:6165:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5584.682306] LustreError: 83086:0:(file.c:6165:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5585.656502] LustreError: 83066:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5585.669129] LustreError: 83066:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff934403fd7000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5585.745604] LustreError: 83092:0:(file.c:6165:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5585.754091] LustreError: 83092:0:(file.c:6165:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5588.062558] LustreError: 83114:0:(file.c:6165:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5588.077773] LustreError: 83114:0:(file.c:6165:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5591.937458] LustreError: 83141:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5594.976723] LustreError: lustre-MDT0000-mdc-ffff934403fd7000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5595.010527] LustreError: 83156:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff934403fd7000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff934404177800) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5595.041417] LustreError: 83156:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5595.942687] LustreError: 83141:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5603.207742] LustreError: lustre-MDT0000-mdc-ffff934403fd7000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5603.215886] LustreError: 83184:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff934403fd7000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff9344185e6900) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5603.232959] LustreError: 83184:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5603.744168] LustreError: 83197:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5607.132558] LustreError: lustre-MDT0000-mdc-ffff934403fd7000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5607.832658] LustreError: 83197:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5610.789401] LustreError: lustre-MDT0000-mdc-ffff934403fd7000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5610.850461] LustreError: 83254:0:(file.c:6165:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5610.856913] LustreError: 83254:0:(file.c:6165:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5612.122667] LustreError: 83225:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff934403fd7000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5619.850486] LustreError: 83306:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5619.856790] LustreError: 83306:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5623.248483] LustreError: lustre-MDT0000-mdc-ffff934403fd7000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5623.323979] LustreError: 83328:0:(file.c:6165:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5623.329614] LustreError: 83328:0:(file.c:6165:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5623.936136] LustreError: 83306:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5623.940584] LustreError: 83306:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5623.952793] LustreError: 83306:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff934403fd7000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5633.582184] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 18:49:24 (1786834164)
[ 5652.515148] LustreError: 81018:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5652.525345] LustreError: 81018:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5656.616224] LustreError: 81018:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5656.625802] LustreError: 81018:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5661.656673] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 18:49:52 (1786834192)
[ 5711.116700] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 18:50:42 (1786834242)
[ 5773.583140] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 18:51:44 (1786834304)
[ 5807.255879] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 18:52:18 (1786834338)
[ 5807.620253] Lustre: Mounted lustre-client - version 2.17.57_3_g928f38d
[ 5931.403595] Lustre: Unmounted lustre-client
[ 5937.347416] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 18:54:28 (1786834468)
[ 5937.984429] LustreError: 87631:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5940.368597] Lustre: lustre-OST0000-osc-ffff934403fd7000: Connection to lustre-OST0000 (at 192.168.202.133@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5940.384813] Lustre: Skipped 16 previous similar messages
[ 5940.395897] LustreError: lustre-OST0000-osc-ffff934403fd7000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5940.411076] LustreError: 87713:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff934403fd7000: namespace resource [0x240000400:0x448d:0x0].0x0 (ffff934418c43b00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5940.430650] LustreError: 87713:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 2 previous similar messages
[ 5940.720211] LustreError: 87631:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5940.729704] Lustre: 2346:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.133@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5946.172993] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 18:54:37 (1786834477)
[ 6003.701304] Lustre: lustre-MDT0000-mdc-ffff934403fd7000: Connection restored to 192.168.202.133@tcp (at 192.168.202.133@tcp)
[ 6003.720171] Lustre: Skipped 16 previous similar messages
[ 6009.346559] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 18:55:40 (1786834540)
[ 6011.070195] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 6012.578837] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 18:55:43 (1786834543)
[ 6074.778641] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6076.104128] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 18:56:47 (1786834607)
[ 6133.728877] Lustre: 90029:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786834610/real 1786834610]  req@ffff93440a6e8a80 x1873625594730368/t0(0) o36->lustre-MDT0000-mdc-ffff934403fd7000@192.168.202.133@tcp:12/10 lens 496/440 e 0 to 1 dl 1786834665 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ln.0' uid:0 gid:0 projid:4294967295
[ 6133.750966] Lustre: 90029:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
[ 6135.161678] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 18:57:46 (1786834666)
[ 6137.043669] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6138.870433] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 18:57:49 (1786834669)
[ 6140.126275] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6141.508825] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 18:57:52 (1786834672)
[ 6176.769657] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 18:58:27 (1786834707)
[ 6188.411144] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6215.664337] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 192.168.202.133@tcp) was lost; in progress operations using this service will fail
[ 6215.677456] LustreError: Skipped 3 previous similar messages
[ 6215.689973] Lustre: Evicted from MGS (at 192.168.202.133@tcp) after server handle changed from 0x208610988174f6a5 to 0x2086109881750ed1
[ 6215.698679] Lustre: Skipped 3 previous similar messages
[ 6217.709199] Lustre: 68224:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.202.133@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6235.033755] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6236.687300] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6245.004782] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 18:59:36 (1786834776)
[ 6246.789769] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6248.010222] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 18:59:39 (1786834779)
[ 6261.742309] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 192.168.202.133@tcp) was lost; in progress operations using this service will fail
[ 6261.771383] Lustre: Evicted from MGS (at 192.168.202.133@tcp) after server handle changed from 0x2086109881750ed1 to 0x208610988175138c
[ 6272.217490] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 19:00:03 (1786834803)
[ 6307.155764] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 19:00:38 (1786834838)
[ 6312.363992] LustreError: lustre-OST0000-osc-ffff934403fd7000: operation ost_setattr to node 192.168.202.133@tcp failed: rc = -107
[ 6312.384633] LustreError: Skipped 7 previous similar messages
[ 6356.076722] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6359.278842] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6441.953319] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 192.168.202.133@tcp) was lost; in progress operations using this service will fail
[ 6441.976961] LustreError: Skipped 1 previous similar message
[ 6452.219367] Lustre: Evicted from MGS (at 192.168.202.133@tcp) after server handle changed from 0x20861098817515d1 to 0x20861098817523e6
[ 6452.237716] Lustre: Skipped 1 previous similar message
[ 6466.598757] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6468.043869] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6495.660085] Lustre: 68224:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.202.133@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6514.996649] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6516.344549] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6631.473447] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 19:06:02 (1786835162)
[ 6640.716394] Lustre: lustre-OST0000-osc-ffff934403fd7000: Connection to lustre-OST0000 (at 192.168.202.133@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6640.735645] Lustre: Skipped 9 previous similar messages
[ 6678.910762] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6681.278963] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6901.206510] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 19:10:32 (1786835432)
[ 7000.036548] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 192.168.202.133@tcp) was lost; in progress operations using this service will fail
[ 7000.041774] LustreError: Skipped 1 previous similar message
[ 7000.072385] Lustre: Evicted from MGS (at 192.168.202.133@tcp) after server handle changed from 0x20861098817527f9 to 0x208610988184cb74
[ 7000.080443] Lustre: Skipped 1 previous similar message
[ 7000.083897] Lustre: MGC192.168.202.133@tcp: Connection restored to 192.168.202.133@tcp (at 192.168.202.133@tcp)
[ 7000.088545] Lustre: Skipped 12 previous similar messages
[ 7024.817683] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 19:12:36 (1786835556)
[ 7025.727717] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7026.776949] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 19:12:38 (1786835558)
[ 7028.007614] LustreError: lustre-MDT0000-mdc-ffff934403fd7000: operation ldlm_enqueue to node 192.168.202.133@tcp failed: rc = -107
[ 7028.022556] LustreError: Skipped 233 previous similar messages
[ 7028.031862] LustreError: lustre-MDT0000-mdc-ffff934403fd7000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7032.589050] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 19:12:44 (1786835564)
[ 7198.961743] LustreError: lustre-OST0000-osc-ffff934403fd7000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7200.054806] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 19:15:31 (1786835731)
[ 7224.800266] Lustre: 2345:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786835736/real 1786835736]  req@ffff93443861f480 x1873625612432640/t0(0) o4->lustre-OST0000-osc-ffff934403fd7000@192.168.202.133@tcp:6/4 lens 4584/448 e 0 to 1 dl 1786835756 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7224.822721] Lustre: 2345:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 26 previous similar messages
[ 7237.915251] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 19:16:09 (1786835769)
[ 7238.691262] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7239.643887] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 19:16:11 (1786835771)
[ 7240.539220] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7241.482948] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 19:16:12 (1786835772)
[ 7269.440408] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 19:16:40 (1786835800)
[ 7301.607338] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 192.168.202.133@tcp) was lost; in progress operations using this service will fail
[ 7301.617470] Lustre: lustre-MDT0000-mdc-ffff934403fd7000: Connection to lustre-MDT0000 (at 192.168.202.133@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7301.626380] Lustre: Evicted from MGS (at 192.168.202.133@tcp) after server handle changed from 0x208610988184cb74 to 0x208610988184f360
[ 7301.626781] Lustre: Skipped 4 previous similar messages
[ 7301.631417] Lustre: Skipped 1 previous similar message
[ 7310.086383] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 19:17:21 (1786835841)
[ 7311.038587] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7312.121896] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 19:17:23 (1786835843)
[ 7313.044694] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7314.087474] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 19:17:25 (1786835845)
[ 7320.592555] Lustre: Unmounted lustre-client
[ 7321.315891] Lustre: Mounted lustre-client - version 2.17.57_3_g928f38d
[ 7323.035942] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7345.523218] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2