[ 1840.708435] LustreError: lustre-OST0000-osc-ffff8a1c11da8800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1840.739938] Lustre: 2365:0:(llite_lib.c:4198:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.132@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1848.971896] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 22:27:26 (1781231246)
[ 1852.677081] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1854.975555] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 22:27:31 (1781231251)
[ 1857.233102] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1859.527963] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 22:27:36 (1781231256)
[ 1863.849077] LustreError: lustre-MDT0000-mdc-ffff8a1c11da8800: operation mds_reint to node 192.168.201.132@tcp failed: rc = -19
[ 1863.868578] LustreError: Skipped 1 previous similar message
[ 1884.139140] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 192.168.201.132@tcp) was lost; in progress operations using this service will fail
[ 1884.185974] Lustre: Evicted from MGS (at 192.168.201.132@tcp) after server handle changed from 0xf56af5eb43fc50d6 to 0xf56af5eb43fc6711
[ 1989.501661] LustreError: lustre-MDT0000-mdc-ffff8a1c11da8800: operation ldlm_enqueue to node 192.168.201.132@tcp failed: rc = -19
[ 1989.512536] LustreError: Skipped 2 previous similar messages
[ 2008.024363] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 192.168.201.132@tcp) was lost; in progress operations using this service will fail
[ 2008.050938] Lustre: Evicted from MGS (at 192.168.201.132@tcp) after server handle changed from 0xf56af5eb43fc6711 to 0xf56af5eb43fe79d6
[ 2009.770043] Lustre: 15880:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.201.132@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2035.828727] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 22:30:32 (1781231432)
[ 2036.575820] Lustre: *** cfs_fail_loc=305, val=0***
[ 2036.579111] Lustre: Skipped 2 previous similar messages
[ 2056.983924] LustreError: lustre-OST0001-osc-ffff8a1c11da8800: operation ost_connect to node 192.168.201.132@tcp failed: rc = -75
[ 2082.784250] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 192.168.201.132@tcp) was lost; in progress operations using this service will fail
[ 2082.820754] Lustre: Evicted from MGS (at 192.168.201.132@tcp) after server handle changed from 0xf56af5eb43fe79d6 to 0xf56af5eb43fe7f55
[ 2095.268304] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2097.092589] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2106.595986] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 22:31:43 (1781231503)
[ 2118.646838] Lustre: lustre-MDT0000-mdc-ffff8a1c11da8800: Connection to lustre-MDT0000 (at 192.168.201.132@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2118.648035] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 192.168.201.132@tcp) was lost; in progress operations using this service will fail
[ 2118.663360] Lustre: Skipped 12 previous similar messages
[ 2118.686474] Lustre: Evicted from MGS (at 192.168.201.132@tcp) after server handle changed from 0xf56af5eb43fe7f55 to 0xf56af5eb43fe82ea
[ 2118.694405] Lustre: MGC192.168.201.132@tcp: Connection restored to 192.168.201.132@tcp (at 192.168.201.132@tcp)
[ 2118.705451] Lustre: Skipped 14 previous similar messages
[ 2120.586249] LustreError: lustre-MDT0000-mdc-ffff8a1c11da8800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2141.035945] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 22:32:18 (1781231538)
[ 2154.477842] LustreError: lustre-OST0000-osc-ffff8a1c11da8800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2171.691555] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 22:32:48 (1781231568)
[ 2185.036606] LustreError: lustre-MDT0000-mdc-ffff8a1c11da8800: operation mds_reint to node 192.168.201.132@tcp failed: rc = -19
[ 2204.652947] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 192.168.201.132@tcp) was lost; in progress operations using this service will fail
[ 2204.671092] Lustre: Evicted from MGS (at 192.168.201.132@tcp) after server handle changed from 0xf56af5eb43fe82ea to 0xf56af5eb43fedb66
[ 2206.660237] Lustre: 15880:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.201.132@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2231.543617] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2233.563575] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2318.304317] Lustre: 2368:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781231700/real 1781231700]  req@ffff8a1c06449880 x1867754446443264/t0(0) o400->MGC192.168.201.132@tcp@192.168.201.132@tcp:26/25 lens 224/224 e 0 to 1 dl 1781231716 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2318.349306] Lustre: 2368:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
[ 2318.358394] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 192.168.201.132@tcp) was lost; in progress operations using this service will fail
[ 2318.397152] Lustre: Evicted from MGS (at 192.168.201.132@tcp) after server handle changed from 0xf56af5eb43fedb66 to 0xf56af5eb4400b2d5
[ 2342.941695] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2345.477797] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2428.897408] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 192.168.201.132@tcp) was lost; in progress operations using this service will fail
[ 2439.092812] Lustre: Evicted from MGS (at 192.168.201.132@tcp) after server handle changed from 0xf56af5eb4400b2d5 to 0xf56af5eb44025acc
[ 2457.333037] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2460.619658] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2490.520030] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 22:38:07 (1781231887)
[ 2495.423496] LustreError: lustre-MDT0000-mdc-ffff8a1c11da8800: operation ldlm_enqueue to node 192.168.201.132@tcp failed: rc = -19
[ 2495.443548] LustreError: Skipped 2 previous similar messages
[ 2515.424653] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 192.168.201.132@tcp) was lost; in progress operations using this service will fail
[ 2524.610878] Lustre: Evicted from MGS (at 192.168.201.132@tcp) after server handle changed from 0xf56af5eb44025acc to 0xf56af5eb44032229
[ 2525.348566] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2557.984773] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2587.566849] Lustre: 15880:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.201.132@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2607.819727] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2643.831265] Lustre: 15880:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.201.132@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2658.824553] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2701.795326] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 192.168.201.132@tcp) was lost; in progress operations using this service will fail
[ 2701.800168] LustreError: Skipped 3 previous similar messages
[ 2711.984872] Lustre: Evicted from MGS (at 192.168.201.132@tcp) after server handle changed from 0xf56af5eb440395da to 0xf56af5eb44040ae9
[ 2711.995886] Lustre: Skipped 3 previous similar messages
[ 2713.032714] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2721.374228] Lustre: lustre-MDT0000-mdc-ffff8a1c11da8800: Connection restored to 192.168.201.132@tcp (at 192.168.201.132@tcp)
[ 2721.382187] Lustre: Skipped 16 previous similar messages
[ 2742.716947] Lustre: lustre-MDT0000-mdc-ffff8a1c11da8800: Connection to lustre-MDT0000 (at 192.168.201.132@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2742.733296] Lustre: Skipped 8 previous similar messages
[ 2773.205387] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2829.418810] Lustre: 15880:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.201.132@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2870.603401] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 22:44:27 (1781232267)
[ 2924.526329] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2926.847996] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3215.967065] LustreError: lustre-OST0000-osc-ffff8a1c11da8800: operation ldlm_enqueue to node 192.168.201.132@tcp failed: rc = -19
[ 3215.973764] LustreError: Skipped 8 previous similar messages
[ 3258.278457] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3260.946204] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3548.424271] Lustre: lustre-OST0000-osc-ffff8a1c11da8800: Connection to lustre-OST0000 (at 192.168.201.132@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3548.443835] Lustre: Skipped 3 previous similar messages
[ 3570.047756] Lustre: lustre-OST0000-osc-ffff8a1c11da8800: Connection restored to 192.168.201.132@tcp (at 192.168.201.132@tcp)
[ 3570.059989] Lustre: Skipped 6 previous similar messages
[ 3589.239864] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3591.723653] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3844.268583] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 23:00:40 (1781233240)
[ 3860.960349] Lustre: 50195:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781233243/real 1781233243]  req@ffff8a1c18026300 x1867754468762368/t0(0) o101->lustre-MDT0000-mdc-ffff8a1c11da8800@192.168.201.132@tcp:12/10 lens 576/1152 e 0 to 1 dl 1781233259 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3860.983751] Lustre: 50195:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
[ 3869.316855] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 23:01:06 (1781233266)
[ 3894.058427] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 23:01:31 (1781233291)
[ 3919.967956] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 23:01:56 (1781233316)
[ 3920.396392] Lustre: Mounted lustre-client
[ 3951.083097] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 192.168.201.132@tcp) was lost; in progress operations using this service will fail
[ 3951.116240] LustreError: Skipped 2 previous similar messages
[ 3951.138199] Lustre: Evicted from MGS (at 192.168.201.132@tcp) after server handle changed from 0xf56af5eb4404eaf0 to 0xf56af5eb441ac5cc
[ 3951.154513] Lustre: Skipped 2 previous similar messages
[ 3952.096085] Lustre: 2368:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781233334/real 1781233334]  req@ffff8a1c104c4a80 x1867754468781056/t0(0) o400->lustre-MDT0000-mdc-ffff8a1c11da8800@192.168.201.132@tcp:12/10 lens 224/224 e 0 to 1 dl 1781233350 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3952.107655] Lustre: 2368:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 3963.262844] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3965.297274] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3967.631271] Lustre: Unmounted lustre-client
[ 3974.530381] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 23:02:51 (1781233371)
[ 4111.840203] Lustre: 2367:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781233494/real 1781233494]  req@ffff8a1c18026d80 x1867754468814464/t0(0) o4->lustre-OST0000-osc-ffff8a1c11da8800@192.168.201.132@tcp:6/4 lens 488/448 e 0 to 1 dl 1781233510 ref 2 fl Rpc:XQr/602/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4111.885220] Lustre: 2367:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 68 previous similar messages
[ 4159.968399] Lustre: lustre-OST0000-osc-ffff8a1c11da8800: Connection to lustre-OST0000 (at 192.168.201.132@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4159.992704] Lustre: Skipped 15 previous similar messages
[ 4176.173876] Lustre: lustre-OST0000-osc-ffff8a1c11da8800: Connection restored to 192.168.201.132@tcp (at 192.168.201.132@tcp)
[ 4176.181100] Lustre: Skipped 17 previous similar messages
[ 4268.954940] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 23:07:46 (1781233666)
[ 4318.056775] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 23:08:34 (1781233714)
[ 4321.129494] Lustre: Unmounted lustre-client
[ 4341.279322] Lustre: Mounted lustre-client
[ 4347.310456] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 23:09:04 (1781233744)
[ 4347.546302] LustreError: 56090:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4348.594237] LustreError: 56090:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4348.670581] Lustre: *** cfs_fail_loc=305, val=0***
[ 4374.344960] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 23:09:30 (1781233770)
[ 4374.869789] Lustre: Mounted lustre-client
[ 4378.070907] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4388.428515] Lustre: Unmounted lustre-client
[ 4395.478855] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 23:09:52 (1781233792)
[ 4469.567053] LustreError: lustre-MDT0000-mdc-ffff8a1c06396000: operation mds_reint to node 192.168.201.132@tcp failed: rc = -19
[ 4469.570571] LustreError: Skipped 1 previous similar message
[ 4487.648154] Lustre: 2366:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781233870/real 1781233870]  req@ffff8a1c19392300 x1867754471491968/t0(0) o400->MGC192.168.201.132@tcp@192.168.201.132@tcp:26/25 lens 224/224 e 0 to 1 dl 1781233886 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4487.672048] Lustre: 2366:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 75 previous similar messages
[ 4487.681253] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 192.168.201.132@tcp) was lost; in progress operations using this service will fail
[ 4487.711598] Lustre: Evicted from MGS (at 192.168.201.132@tcp) after server handle changed from 0xf56af5eb441adff0 to 0xf56af5eb441d9598
[ 4645.480882] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 23:14:02 (1781234042)
[ 4651.467517] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4665.841145] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 192.168.201.132@tcp) was lost; in progress operations using this service will fail
[ 4665.871486] Lustre: Evicted from MGS (at 192.168.201.132@tcp) after server handle changed from 0xf56af5eb441d9598 to 0xf56af5eb44225c0d
[ 4676.153053] LustreError: lustre-MDT0000-mdc-ffff8a1c06396000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4685.623952] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 23:14:42 (1781234082)
[ 4686.057527] Lustre: Mounted lustre-client
[ 4693.426981] LustreError: lustre-OST0000-osc-ffff8a1c06396000: operation ldlm_enqueue to node 192.168.201.132@tcp failed: rc = -107
[ 4693.446075] LustreError: lustre-OST0000-osc-ffff8a1c06396000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4693.461978] LustreError: 59788:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-OST0000-osc-ffff8a1c06396000: namespace resource [0x240000400:0x3fc3:0x0].0x0 (ffff8a1c131f0600) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4704.562052] Lustre: Unmounted lustre-client
[ 4710.735377] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 23:15:07 (1781234107)
[ 4717.322682] LustreError: lustre-MDT0000-mdc-ffff8a1c06396000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4717.346541] LustreError: 60462:0:(file.c:6103:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4725.227680] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 23:15:21 (1781234121)
[ 4725.429681] LustreError: 61166:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4729.352895] LustreError: lustre-MDT0000-mdc-ffff8a1c06396000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4729.363615] LustreError: 61182:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4729.369215] LustreError: 61166:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=1065
[ 4729.377568] LustreError: 61166:0:(import.c:719:ptlrpc_connect_import_locked()) already connecting
[ 4729.435970] LustreError: 61187:0:(file.c:6103:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4730.521634] LustreError: 61194:0:(file.c:6103:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4730.532730] LustreError: 61194:0:(file.c:6103:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4732.912220] LustreError: 61216:0:(file.c:6103:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4732.926285] LustreError: 61216:0:(file.c:6103:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4737.664388] LustreError: 61261:0:(file.c:6103:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4737.673525] LustreError: 61261:0:(file.c:6103:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4747.297714] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 23:15:44 (1781234144)
[ 4784.616945] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4786.655652] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4797.299162] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 23:16:33 (1781234193)
[ 4806.126263] Lustre: lustre-OST0000-osc-ffff8a1c06396000: Connection to lustre-OST0000 (at 192.168.201.132@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4806.152598] Lustre: Skipped 17 previous similar messages
[ 4822.548898] Lustre: lustre-OST0000-osc-ffff8a1c06396000: Connection restored to 192.168.201.132@tcp (at 192.168.201.132@tcp)
[ 4822.559808] Lustre: Skipped 16 previous similar messages
[ 4836.067653] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4838.085578] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4849.285972] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 23:17:26 (1781234246)
[ 4910.580959] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4912.364986] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4921.082603] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 23:18:38 (1781234318)
[ 4962.923708] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4965.080757] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4972.625905] Lustre: Unmounted lustre-client
[ 5023.427304] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5026.158115] Lustre: Mounted lustre-client
[ 5033.289133] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 23:20:30 (1781234430)
[ 5036.118601] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5038.250920] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 23:20:35 (1781234435)
[ 5073.789303] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 23:21:10 (1781234470)
[ 5075.793305] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5077.667565] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 23:21:14 (1781234474)
[ 5079.127669] Lustre: *** cfs_fail_loc=805, val=0***
[ 5079.169588] Lustre: Mounted lustre-client
[ 5084.166245] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5105.149606] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 192.168.201.132@tcp) was lost; in progress operations using this service will fail
[ 5105.195736] Lustre: Evicted from MGS (at 192.168.201.132@tcp) after server handle changed from 0xf56af5eb44226862 to 0xf56af5eb44226b79
[ 5106.152354] Lustre: 2367:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781234488/real 1781234488]  req@ffff8a1c104c5880 x1867754473538944/t0(0) o400->lustre-MDT0000-mdc-ffff8a1c044f7000@192.168.201.132@tcp:12/10 lens 224/224 e 0 to 1 dl 1781234504 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5106.194725] Lustre: 2367:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5114.461334] Lustre: Unmounted lustre-client
[ 5123.044609] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 23:22:00 (1781234520)
[ 5150.275301] LustreError: 2364:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff8a1c05d5bb80 x1867754473549824/t90194313220(90194313220) o101->lustre-MDT0000-mdc-ffff8a1c04da3000@192.168.201.132@tcp:12/10 lens 648/608 e 0 to 0 dl 1781234564 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5156.322307] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5158.111670] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5166.359411] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 23:22:43 (1781234563)
[ 5167.820696] LustreError: lustre-OST0000-osc-ffff8a1c04da3000: operation ost_write to node 192.168.201.132@tcp failed: rc = -107
[ 5167.828520] LustreError: Skipped 2 previous similar messages
[ 5167.836351] LustreError: lustre-OST0000-osc-ffff8a1c04da3000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5167.845519] Lustre: 2367:0:(llite_lib.c:4198:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.132@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5167.867075] LustreError: 72150:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-OST0000-osc-ffff8a1c04da3000: namespace resource [0x240000400:0x4022:0x0].0x0 (ffff8a1c06949000) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5175.932664] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 23:22:52 (1781234572)
[ 5177.355563] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5178.812293] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 23:22:56 (1781234576)
[ 5180.401652] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5182.496764] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 23:22:59 (1781234579)
[ 5183.950556] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5185.520937] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 23:23:02 (1781234582)
[ 5186.898906] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5188.648865] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 23:23:05 (1781234585)
[ 5190.068332] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5191.477314] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 23:23:08 (1781234588)
[ 5192.983898] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5194.565346] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 23:23:11 (1781234591)
[ 5195.929133] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5197.825948] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 23:23:14 (1781234594)
[ 5199.317254] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5200.807587] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 23:23:17 (1781234597)
[ 5202.365540] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5203.722180] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 23:23:21 (1781234601)
[ 5205.274563] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5206.709813] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 23:23:24 (1781234604)
[ 5208.007957] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5209.516714] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 23:23:26 (1781234606)
[ 5212.034815] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5213.791100] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 23:23:31 (1781234611)
[ 5242.202210] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 23:23:59 (1781234639)
[ 5273.430678] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 23:24:29 (1781234669)
[ 5273.985864] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5285.668325] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 23:24:42 (1781234682)
[ 5286.125747] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5296.107042] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 23:24:53 (1781234693)
[ 5296.461795] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5305.049626] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 23:25:02 (1781234702)
[ 5305.306729] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5313.098805] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 23:25:10 (1781234710)
[ 5313.426068] Lustre: *** cfs_fail_loc=510, val=3***
[ 5321.023499] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 23:25:18 (1781234718)
[ 5321.327474] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5331.132350] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 23:25:28 (1781234728)
[ 5331.449786] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5397.026503] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 23:26:33 (1781234793)
[ 5397.478670] LustreError: 82342:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5400.531635] LustreError: lustre-MDT0000-mdc-ffff8a1c04da3000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5400.541779] LustreError: 82356:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff8a1c04da3000: inode [0x20000a042:0x8:0x0] mdc close failed: rc = -108
[ 5400.564097] LustreError: 82356:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8a1c04da3000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8a1c063cf900) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5401.560148] LustreError: 82342:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5403.684309] LustreError: 82372:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5406.382368] Lustre: lustre-MDT0000-mdc-ffff8a1c04da3000: Connection to lustre-MDT0000 (at 192.168.201.132@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5406.390596] Lustre: Skipped 9 previous similar messages
[ 5406.404550] LustreError: lustre-MDT0000-mdc-ffff8a1c04da3000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5406.434566] LustreError: 82386:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8a1c04da3000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8a1c06949200) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5406.452129] LustreError: 82386:0:(ldlm_resource.c:1180:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5407.777947] LustreError: 82372:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5414.852165] LustreError: lustre-MDT0000-mdc-ffff8a1c04da3000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5415.455485] LustreError: 82428:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5418.382560] LustreError: lustre-MDT0000-mdc-ffff8a1c04da3000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5418.486273] LustreError: 82448:0:(file.c:6103:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5418.502913] LustreError: 82448:0:(file.c:6103:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5419.460426] LustreError: 82428:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5419.466818] LustreError: 82428:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff8a1c04da3000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5419.476585] LustreError: 82428:0:(file.c:252:ll_close_inode_openhandle()) Skipped 3 previous similar messages
[ 5419.749321] LustreError: 82454:0:(file.c:6103:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5419.758405] LustreError: 82454:0:(file.c:6103:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5422.126113] LustreError: 82476:0:(file.c:6103:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5422.140592] LustreError: 82476:0:(file.c:6103:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5425.882739] LustreError: 82503:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5428.998912] LustreError: lustre-MDT0000-mdc-ffff8a1c04da3000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5429.019271] LustreError: 82518:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8a1c04da3000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8a1c063cf100) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5429.056350] LustreError: 82518:0:(ldlm_resource.c:1180:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5429.088665] Lustre: lustre-MDT0000-mdc-ffff8a1c04da3000: Connection restored to 192.168.201.132@tcp (at 192.168.201.132@tcp)
[ 5429.113626] Lustre: Skipped 13 previous similar messages
[ 5429.985254] LustreError: 82503:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5437.317897] LustreError: lustre-MDT0000-mdc-ffff8a1c04da3000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5437.738886] LustreError: 82559:0:(ldlm_flock.c:858:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5440.773996] LustreError: lustre-MDT0000-mdc-ffff8a1c04da3000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5440.801129] LustreError: 82574:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8a1c04da3000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8a1c0688e100) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5440.823858] LustreError: 82574:0:(ldlm_resource.c:1180:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5441.744134] LustreError: 82559:0:(ldlm_flock.c:858:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5444.946684] LustreError: lustre-MDT0000-mdc-ffff8a1c04da3000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5445.044477] LustreError: 82607:0:(file.c:6103:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5445.053607] LustreError: 82607:0:(file.c:6103:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5445.956082] LustreError: 82587:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff8a1c04da3000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5454.106260] LustreError: 82659:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5454.118678] LustreError: 82659:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5457.593564] LustreError: lustre-MDT0000-mdc-ffff8a1c04da3000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5457.700761] LustreError: 82681:0:(file.c:6103:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5457.715652] LustreError: 82681:0:(file.c:6103:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5458.200130] LustreError: 82659:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5458.210438] LustreError: 82659:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5458.231072] LustreError: 82659:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff8a1c04da3000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5470.027230] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 23:27:46 (1781234866)
[ 5488.754550] LustreError: 67660:0:(ldlm_lockd.c:2883:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5488.766219] LustreError: 67660:0:(ldlm_lockd.c:2883:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5492.760171] LustreError: 67660:0:(ldlm_lockd.c:2883:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5492.766892] LustreError: 67660:0:(ldlm_lockd.c:2883:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5500.964867] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 23:28:17 (1781234897)
[ 5553.351889] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 23:29:10 (1781234950)
[ 5619.897884] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 23:30:17 (1781235017)
[ 5653.024100] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 23:30:50 (1781235050)
[ 5653.631992] Lustre: Mounted lustre-client
[ 5776.880993] Lustre: Unmounted lustre-client
[ 5783.420813] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 23:33:00 (1781235180)
[ 5783.885228] LustreError: 86983:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5786.465169] LustreError: lustre-OST0000-osc-ffff8a1c04da3000: operation ost_statfs to node 192.168.201.132@tcp failed: rc = -107
[ 5786.469240] LustreError: Skipped 9 previous similar messages
[ 5786.478815] LustreError: lustre-OST0000-osc-ffff8a1c04da3000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5786.486992] LustreError: 87070:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-OST0000-osc-ffff8a1c04da3000: namespace resource [0x240000400:0x404d:0x0].0x0 (ffff8a1c06949d00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5786.500948] LustreError: 87070:0:(ldlm_resource.c:1180:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5786.728136] LustreError: 86983:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5786.749408] Lustre: 2366:0:(llite_lib.c:4198:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.132@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5792.701719] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 23:33:09 (1781235189)
[ 5851.104164] Lustre: 87649:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781235193/real 1781235193]  req@ffff8a1c06891180 x1867754473717888/t0(0) o101->lustre-MDT0000-mdc-ffff8a1c04da3000@192.168.201.132@tcp:12/10 lens 328/344 e 0 to 1 dl 1781235249 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 5851.146249] Lustre: 87649:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 15 previous similar messages
[ 5857.551862] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 23:34:14 (1781235254)
[ 5858.924910] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 5860.358709] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 23:34:17 (1781235257)
[ 5924.537458] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 5926.299988] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 23:35:23 (1781235323)
[ 5985.815926] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 23:36:22 (1781235382)
[ 5987.848414] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 5989.947730] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 23:36:26 (1781235386)
[ 5991.900491] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 5993.918943] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 23:36:30 (1781235390)
[ 6034.873229] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 23:37:11 (1781235431)
[ 6047.329336] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6072.288962] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 192.168.201.132@tcp) was lost; in progress operations using this service will fail
[ 6072.296279] LustreError: Skipped 2 previous similar messages
[ 6077.414207] Lustre: lustre-MDT0000-mdc-ffff8a1c04da3000: Connection to lustre-MDT0000 (at 192.168.201.132@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6077.444793] Lustre: Skipped 14 previous similar messages
[ 6077.570762] Lustre: lustre-MDT0000-mdc-ffff8a1c04da3000: Connection restored to 192.168.201.132@tcp (at 192.168.201.132@tcp)
[ 6077.576041] Lustre: Skipped 11 previous similar messages
[ 6081.656270] Lustre: Evicted from MGS (at 192.168.201.132@tcp) after server handle changed from 0xf56af5eb442276bd to 0xf56af5eb44229166
[ 6081.679403] Lustre: Skipped 2 previous similar messages
[ 6086.860427] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6088.834213] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6099.993145] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 23:38:16 (1781235496)
[ 6102.724528] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6104.270276] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 23:38:21 (1781235501)
[ 6117.893811] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 192.168.201.132@tcp) was lost; in progress operations using this service will fail
[ 6117.948798] Lustre: Evicted from MGS (at 192.168.201.132@tcp) after server handle changed from 0xf56af5eb44229166 to 0xf56af5eb44229381
[ 6132.816050] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 23:38:49 (1781235529)
[ 6184.193211] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 23:39:41 (1781235581)
[ 6241.152937] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 6247.337041] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6331.875959] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 192.168.201.132@tcp) was lost; in progress operations using this service will fail
[ 6331.887766] LustreError: Skipped 1 previous similar message
[ 6342.141303] Lustre: Evicted from MGS (at 192.168.201.132@tcp) after server handle changed from 0xf56af5eb44229708 to 0xf56af5eb4422a3db
[ 6342.164019] Lustre: Skipped 1 previous similar message
[ 6359.532334] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6362.134814] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6413.279900] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6415.836184] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6453.664172] Lustre: 2366:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781235792/real 1781235792]  req@ffff8a1c189f6680 x1867754480247680/t0(0) o400->lustre-MDT0000-mdc-ffff8a1c04da3000@192.168.201.132@tcp:12/10 lens 224/224 e 0 to 1 dl 1781235852 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6453.713786] Lustre: 2366:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 30 previous similar messages
[ 6555.213460] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 23:45:52 (1781235952)
[ 6566.190616] LustreError: lustre-OST0000-osc-ffff8a1c04da3000: operation ost_setattr to node 192.168.201.132@tcp failed: rc = -19
[ 6566.203299] LustreError: Skipped 38 previous similar messages
[ 6613.746924] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 6617.177573] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6893.552821] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 23:51:30 (1781236290)
[ 7034.358852] Lustre: lustre-MDT0000-mdc-ffff8a1c04da3000: Connection to lustre-MDT0000 (at 192.168.201.132@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7034.374500] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 192.168.201.132@tcp) was lost; in progress operations using this service will fail
[ 7034.391937] Lustre: Skipped 6 previous similar messages
[ 7034.413982] LustreError: Skipped 1 previous similar message
[ 7034.449103] Lustre: Evicted from MGS (at 192.168.201.132@tcp) after server handle changed from 0xf56af5eb4422a707 to 0xf56af5eb44324b23
[ 7034.456803] Lustre: Skipped 1 previous similar message
[ 7034.468248] Lustre: MGC192.168.201.132@tcp: Connection restored to 192.168.201.132@tcp (at 192.168.201.132@tcp)
[ 7034.474037] Lustre: Skipped 9 previous similar messages
[ 7061.984100] Lustre: 2365:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781236422/real 1781236422]  req@ffff8a1c04646300 x1867754491404160/t0(0) o400->lustre-MDT0000-mdc-ffff8a1c04da3000@192.168.201.132@tcp:12/10 lens 224/224 e 0 to 1 dl 1781236460 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7071.805526] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 23:54:28 (1781236468)
[ 7074.172663] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7076.661976] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 23:54:33 (1781236473)
[ 7079.900363] LustreError: lustre-MDT0000-mdc-ffff8a1c04da3000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7088.524880] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 23:54:45 (1781236485)
[ 7255.871342] LustreError: lustre-OST0000-osc-ffff8a1c04da3000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7257.478087] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 23:57:34 (1781236654)
[ 7302.103563] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 23:58:19 (1781236699)
[ 7303.792768] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7305.816467] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 23:58:22 (1781236702)
[ 7307.338141] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7309.118647] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 23:58:26 (1781236706)
[ 7345.890581] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 23:59:02 (1781236742)
[ 7389.169034] LustreError: MGC192.168.201.132@tcp: Connection to MGS (at 192.168.201.132@tcp) was lost; in progress operations using this service will fail
[ 7389.195541] Lustre: Evicted from MGS (at 192.168.201.132@tcp) after server handle changed from 0xf56af5eb44324b23 to 0xf56af5eb44327298
[ 7389.209555] Lustre: Skipped 1 previous similar message
[ 7402.621663] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 23:59:59 (1781236799)
[ 7404.524302] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7406.502850] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 00:00:03 (1781236803)
[ 7408.640703] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7410.712932] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 00:00:07 (1781236807)
[ 7428.794068] Lustre: Unmounted lustre-client
[ 7430.694250] Lustre: Mounted lustre-client
[ 7433.914880] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7464.146693] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2