[ 1977.811214] LustreError: lustre-OST0000-osc-ffff9424081b9000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1977.837468] Lustre: 2359:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.155@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1984.919450] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 22:46:02 (1787625962)
[ 1987.428882] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1988.985638] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 22:46:06 (1787625966)
[ 1990.895430] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1992.321657] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 22:46:10 (1787625970)
[ 1995.931315] LustreError: lustre-MDT0000-mdc-ffff9424081b9000: operation mds_close to node 192.168.202.155@tcp failed: rc = -19
[ 1995.943495] LustreError: Skipped 1 previous similar message
[ 2013.152823] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 192.168.202.155@tcp) was lost; in progress operations using this service will fail
[ 2023.346325] Lustre: Evicted from MGS (at 192.168.202.155@tcp) after server handle changed from 0xf0a4e92a419924c to 0xf0a4e92a419acee
[ 2125.808887] LustreError: lustre-MDT0000-mdc-ffff9424081b9000: operation mds_reint to node 192.168.202.155@tcp failed: rc = -19
[ 2125.820839] LustreError: Skipped 2 previous similar messages
[ 2144.232656] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 192.168.202.155@tcp) was lost; in progress operations using this service will fail
[ 2144.261615] Lustre: Evicted from MGS (at 192.168.202.155@tcp) after server handle changed from 0xf0a4e92a419acee to 0xf0a4e92a41b8e51
[ 2146.223876] Lustre: 15944:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.155@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2171.679642] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 22:49:09 (1787626149)
[ 2172.000212] Lustre: *** cfs_fail_loc=305, val=0***
[ 2172.004633] Lustre: Skipped 2 previous similar messages
[ 2215.904290] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 192.168.202.155@tcp) was lost; in progress operations using this service will fail
[ 2215.943133] Lustre: lustre-MDT0000-mdc-ffff9424081b9000: Connection to lustre-MDT0000 (at 192.168.202.155@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2215.977396] Lustre: Skipped 11 previous similar messages
[ 2215.978055] Lustre: Evicted from MGS (at 192.168.202.155@tcp) after server handle changed from 0xf0a4e92a41b8e51 to 0xf0a4e92a41b93c2
[ 2216.031675] Lustre: MGC192.168.202.155@tcp: Connection restored to 192.168.202.155@tcp (at 192.168.202.155@tcp)
[ 2216.051648] Lustre: Skipped 13 previous similar messages
[ 2231.963578] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2233.364257] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2242.263569] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 22:50:19 (1787626219)
[ 2256.900333] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 192.168.202.155@tcp) was lost; in progress operations using this service will fail
[ 2256.930630] Lustre: Evicted from MGS (at 192.168.202.155@tcp) after server handle changed from 0xf0a4e92a41b93c2 to 0xf0a4e92a41b9899
[ 2257.172766] LustreError: lustre-MDT0000-mdc-ffff9424081b9000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2284.995501] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 22:51:03 (1787626263)
[ 2302.970069] LustreError: lustre-OST0000-osc-ffff9424081b9000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2327.856572] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 22:51:44 (1787626304)
[ 2341.639151] LustreError: lustre-MDT0000-mdc-ffff9424081b9000: operation mds_reint to node 192.168.202.155@tcp failed: rc = -19
[ 2360.289801] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 192.168.202.155@tcp) was lost; in progress operations using this service will fail
[ 2370.545960] Lustre: Evicted from MGS (at 192.168.202.155@tcp) after server handle changed from 0xf0a4e92a41b9899 to 0xf0a4e92a41bedcd
[ 2391.727610] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2394.544743] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2477.030152] Lustre: 2358:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787626440/real 1787626440]  req@ffff94240ab0b800 x1874459674550656/t0(0) o400->MGC192.168.202.155@tcp@192.168.202.155@tcp:26/25 lens 224/224 e 0 to 1 dl 1787626456 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2477.046661] Lustre: 2358:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 12 previous similar messages
[ 2477.055825] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 192.168.202.155@tcp) was lost; in progress operations using this service will fail
[ 2487.283991] Lustre: Evicted from MGS (at 192.168.202.155@tcp) after server handle changed from 0xf0a4e92a41bedcd to 0xf0a4e92a41dc99c
[ 2504.529827] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2507.101387] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2572.756602] LustreError: lustre-MDT0000-mdc-ffff9424081b9000: operation mds_reint to node 192.168.202.155@tcp failed: rc = -19
[ 2572.764515] LustreError: Skipped 2 previous similar messages
[ 2589.664428] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 192.168.202.155@tcp) was lost; in progress operations using this service will fail
[ 2599.850316] Lustre: Evicted from MGS (at 192.168.202.155@tcp) after server handle changed from 0xf0a4e92a41dc99c to 0xf0a4e92a41f9ef7
[ 2618.334079] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2621.344403] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2651.034735] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 22:57:08 (1787626628)
[ 2674.677713] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 192.168.202.155@tcp) was lost; in progress operations using this service will fail
[ 2674.691759] Lustre: Evicted from MGS (at 192.168.202.155@tcp) after server handle changed from 0xf0a4e92a41f9ef7 to 0xf0a4e92a42075dc
[ 2683.524704] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2715.050336] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2750.322092] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2789.424097] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2832.353658] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 192.168.202.155@tcp) was lost; in progress operations using this service will fail
[ 2832.365591] LustreError: Skipped 3 previous similar messages
[ 2832.396890] Lustre: Evicted from MGS (at 192.168.202.155@tcp) after server handle changed from 0xf0a4e92a4213399 to 0xf0a4e92a421865e
[ 2832.405321] Lustre: Skipped 3 previous similar messages
[ 2832.411814] Lustre: MGC192.168.202.155@tcp: Connection restored to 192.168.202.155@tcp (at 192.168.202.155@tcp)
[ 2832.417403] Lustre: Skipped 18 previous similar messages
[ 2841.457443] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2871.451393] LustreError: lustre-MDT0000-mdc-ffff9424081b9000: operation mds_reint to node 192.168.202.155@tcp failed: rc = -19
[ 2871.463328] LustreError: Skipped 9 previous similar messages
[ 2871.473518] Lustre: lustre-MDT0000-mdc-ffff9424081b9000: Connection to lustre-MDT0000 (at 192.168.202.155@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2871.502336] Lustre: Skipped 10 previous similar messages
[ 2909.552439] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2997.466191] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 23:02:55 (1787626975)
[ 3053.210183] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3055.570214] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3386.286046] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3388.955921] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3676.176163] LustreError: lustre-OST0000-osc-ffff9424081b9000: operation ldlm_enqueue to node 192.168.202.155@tcp failed: rc = -19
[ 3676.202026] LustreError: Skipped 3 previous similar messages
[ 3676.207510] Lustre: lustre-OST0000-osc-ffff9424081b9000: Connection to lustre-OST0000 (at 192.168.202.155@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3676.216696] Lustre: Skipped 3 previous similar messages
[ 3702.488903] Lustre: lustre-OST0000-osc-ffff9424081b9000: Connection restored to 192.168.202.155@tcp (at 192.168.202.155@tcp)
[ 3702.508501] Lustre: Skipped 7 previous similar messages
[ 3724.561984] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3727.786650] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3972.920653] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 23:19:10 (1787627950)
[ 3989.984217] Lustre: 50581:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787627953/real 1787627953]  req@ffff94240b39ad80 x1874459696720000/t0(0) o101->lustre-MDT0000-mdc-ffff9424081b9000@192.168.202.155@tcp:12/10 lens 576/1152 e 0 to 1 dl 1787627969 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3990.018349] Lustre: 50581:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
[ 3999.253693] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 23:19:36 (1787627976)
[ 4026.429563] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 23:20:03 (1787628003)
[ 4053.168294] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 23:20:30 (1787628030)
[ 4053.741824] Lustre: Mounted lustre-client - version 2.17.57_80_ga3b037e
[ 4085.728776] Lustre: 2358:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787628048/real 1787628048]  req@ffff94241004f800 x1874459696739840/t0(0) o400->lustre-MDT0000-mdc-ffff9424081b9000@192.168.202.155@tcp:12/10 lens 224/224 e 0 to 1 dl 1787628064 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4085.818617] Lustre: 2358:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 4085.824313] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 192.168.202.155@tcp) was lost; in progress operations using this service will fail
[ 4085.831195] LustreError: Skipped 2 previous similar messages
[ 4094.972311] Lustre: Evicted from MGS (at 192.168.202.155@tcp) after server handle changed from 0xf0a4e92a422778a to 0xf0a4e92a437d626
[ 4094.986083] Lustre: Skipped 2 previous similar messages
[ 4096.438208] Lustre: 15944:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.155@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 4116.200494] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4117.906498] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4120.094745] Lustre: Unmounted lustre-client
[ 4129.136813] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 23:21:46 (1787628106)
[ 4237.280265] Lustre: 2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787628200/real 1787628200]  req@ffff94240abf8380 x1874459696775936/t0(0) o4->lustre-OST0000-osc-ffff9424081b9000@192.168.202.155@tcp:6/4 lens 488/448 e 0 to 1 dl 1787628216 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4237.313383] Lustre: 2359:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 49 previous similar messages
[ 4285.216472] Lustre: lustre-OST0000-osc-ffff9424081b9000: Connection to lustre-OST0000 (at 192.168.202.155@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4285.237118] Lustre: Skipped 13 previous similar messages
[ 4317.165480] Lustre: lustre-OST0000-osc-ffff9424081b9000: Connection restored to 192.168.202.155@tcp (at 192.168.202.155@tcp)
[ 4317.182328] Lustre: Skipped 16 previous similar messages
[ 4429.854749] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 23:26:46 (1787628406)
[ 4479.640074] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 23:27:37 (1787628457)
[ 4482.359745] Lustre: Unmounted lustre-client
[ 4506.124580] Lustre: Mounted lustre-client - version 2.17.57_80_ga3b037e
[ 4512.412973] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 23:28:10 (1787628490)
[ 4512.826154] LustreError: 56512:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4513.883212] LustreError: 56512:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4514.093527] Lustre: *** cfs_fail_loc=305, val=0***
[ 4537.277291] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 23:28:35 (1787628515)
[ 4537.757610] Lustre: Mounted lustre-client - version 2.17.57_80_ga3b037e
[ 4539.775630] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4550.143766] Lustre: Unmounted lustre-client
[ 4559.911947] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 23:28:56 (1787628536)
[ 4660.137367] LustreError: lustre-MDT0000-mdc-ffff942409e74800: operation mds_reint to node 192.168.202.155@tcp failed: rc = -19
[ 4677.088624] Lustre: 2360:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787628640/real 1787628640]  req@ffff942410600380 x1874459699450368/t0(0) o400->MGC192.168.202.155@tcp@192.168.202.155@tcp:26/25 lens 224/224 e 0 to 1 dl 1787628656 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4677.104720] Lustre: 2360:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 86 previous similar messages
[ 4677.111662] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 192.168.202.155@tcp) was lost; in progress operations using this service will fail
[ 4687.334644] Lustre: Evicted from MGS (at 192.168.202.155@tcp) after server handle changed from 0xf0a4e92a437f433 to 0xf0a4e92a43aa662
[ 4688.750869] Lustre: 55910:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.155@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 4832.625205] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 23:33:30 (1787628810)
[ 4838.015469] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4850.159623] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 192.168.202.155@tcp) was lost; in progress operations using this service will fail
[ 4850.189432] Lustre: Evicted from MGS (at 192.168.202.155@tcp) after server handle changed from 0xf0a4e92a43aa662 to 0xf0a4e92a43f726b
[ 4851.823043] LustreError: lustre-MDT0000-mdc-ffff942409e74800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4865.307642] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 23:34:03 (1787628843)
[ 4865.618806] Lustre: Mounted lustre-client - version 2.17.57_80_ga3b037e
[ 4872.875293] LustreError: lustre-OST0000-osc-ffff942409e74800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4884.141967] Lustre: Unmounted lustre-client
[ 4891.007337] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 23:34:28 (1787628868)
[ 4894.500449] Lustre: lustre-MDT0000-mdc-ffff942409e74800: Connection to lustre-MDT0000 (at 192.168.202.155@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4894.510237] Lustre: Skipped 15 previous similar messages
[ 4897.481857] LustreError: lustre-MDT0000-mdc-ffff942409e74800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4897.504843] LustreError: 60904:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4904.818533] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 23:34:42 (1787628882)
[ 4905.083811] LustreError: 61607:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4908.991700] LustreError: lustre-MDT0000-mdc-ffff942409e74800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4909.007378] LustreError: 61624:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4909.025349] LustreError: 61607:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=1065
[ 4909.045341] LustreError: 61607:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting
[ 4909.056675] LustreError: 61629:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4910.132918] LustreError: 61635:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4910.162726] LustreError: 61635:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4912.473185] LustreError: 61658:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4912.488203] LustreError: 61658:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4917.137856] LustreError: 61702:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4917.145858] LustreError: 61702:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4919.270303] Lustre: lustre-MDT0000-mdc-ffff942409e74800: Connection restored to 192.168.202.155@tcp (at 192.168.202.155@tcp)
[ 4919.289083] Lustre: Skipped 17 previous similar messages
[ 4928.320509] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 23:35:05 (1787628905)
[ 4969.306965] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4970.533193] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4979.442350] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 23:35:57 (1787628957)
[ 5015.611603] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5017.014586] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5026.566173] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 23:36:44 (1787629004)
[ 5092.444538] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5094.021925] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5103.877926] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 23:38:00 (1787629080)
[ 5145.430845] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5147.277761] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5153.654710] Lustre: Unmounted lustre-client
[ 5206.670920] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5209.088905] Lustre: Mounted lustre-client - version 2.17.57_80_ga3b037e
[ 5215.210398] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 23:39:52 (1787629192)
[ 5218.858207] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5221.466455] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 23:39:58 (1787629198)
[ 5251.658697] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 23:40:28 (1787629228)
[ 5253.217431] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5255.443885] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 23:40:32 (1787629232)
[ 5256.832383] Lustre: *** cfs_fail_loc=805, val=0***
[ 5256.884496] Lustre: Mounted lustre-client - version 2.17.57_80_ga3b037e
[ 5261.847601] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5283.810311] Lustre: 2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787629246/real 1787629246]  req@ffff94241078f480 x1874459701504000/t0(0) o400->lustre-MDT0000-mdc-ffff942411353800@192.168.202.155@tcp:12/10 lens 224/224 e 0 to 1 dl 1787629262 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5283.855428] Lustre: 2359:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5283.868408] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 192.168.202.155@tcp) was lost; in progress operations using this service will fail
[ 5283.899908] Lustre: Evicted from MGS (at 192.168.202.155@tcp) after server handle changed from 0xf0a4e92a43f7ec0 to 0xf0a4e92a43f80db
[ 5285.817740] Lustre: 68269:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.155@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 5302.978737] Lustre: Unmounted lustre-client
[ 5313.547313] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 23:41:30 (1787629290)
[ 5345.585744] Lustre: 68269:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.155@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 5356.645869] LustreError: 2357:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff9424105f7800 x1874459701517568/t94489280516(94489280516) o101->lustre-MDT0000-mdc-ffff942411353800@192.168.202.155@tcp:12/10 lens 648/608 e 0 to 0 dl 1787629351 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5370.599684] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5372.832982] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5381.016692] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 23:42:39 (1787629359)
[ 5382.774136] LustreError: lustre-OST0000-osc-ffff942411353800: operation ost_write to node 192.168.202.155@tcp failed: rc = -107
[ 5382.785012] LustreError: Skipped 3 previous similar messages
[ 5382.815308] LustreError: lustre-OST0000-osc-ffff942411353800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5382.830187] Lustre: 2358:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.155@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5382.870054] LustreError: 72812:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff942411353800: namespace resource [0x240000400:0x3f62:0x0].0x0 (ffff94240660b200) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5390.208288] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 23:42:48 (1787629368)
[ 5391.779332] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5393.461746] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 23:42:51 (1787629371)
[ 5394.680394] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5396.052753] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 23:42:54 (1787629374)
[ 5397.336604] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5399.914238] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 23:42:56 (1787629376)
[ 5402.159497] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5404.530506] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 23:43:01 (1787629381)
[ 5406.965901] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5409.193454] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 23:43:06 (1787629386)
[ 5411.070561] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5414.040106] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 23:43:10 (1787629390)
[ 5416.403431] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5419.637950] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 23:43:16 (1787629396)
[ 5421.553594] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5424.208094] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 23:43:21 (1787629401)
[ 5426.843601] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5429.114368] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 23:43:26 (1787629406)
[ 5431.987936] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5434.251388] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 23:43:31 (1787629411)
[ 5436.168730] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5438.237537] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 23:43:35 (1787629415)
[ 5440.539789] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5441.894102] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 23:43:39 (1787629419)
[ 5472.904649] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 23:44:10 (1787629450)
[ 5501.859655] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 23:44:39 (1787629479)
[ 5502.281448] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5512.116500] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 23:44:49 (1787629489)
[ 5512.402580] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5522.636639] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 23:45:00 (1787629500)
[ 5523.088712] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5532.403866] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 23:45:09 (1787629509)
[ 5532.779742] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5542.345719] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 23:45:20 (1787629520)
[ 5542.764675] Lustre: *** cfs_fail_loc=510, val=3***
[ 5552.014414] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 23:45:29 (1787629529)
[ 5552.417994] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5562.587236] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 23:45:40 (1787629540)
[ 5563.190065] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5622.757488] Lustre: lustre-OST0000-osc-ffff942411353800: Connection to lustre-OST0000 (at 192.168.202.155@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5622.785786] Lustre: Skipped 12 previous similar messages
[ 5629.818902] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 23:46:47 (1787629607)
[ 5630.371564] LustreError: 83027:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5633.135741] LustreError: lustre-MDT0000-mdc-ffff942411353800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5633.154890] LustreError: 83042:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff942411353800: inode [0x20000a042:0xb:0x0] mdc close failed: rc = -108
[ 5633.170371] LustreError: 83042:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff942411353800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff942407021000) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5633.198540] Lustre: lustre-MDT0000-mdc-ffff942411353800: Connection restored to 192.168.202.155@tcp (at 192.168.202.155@tcp)
[ 5633.206675] Lustre: Skipped 10 previous similar messages
[ 5634.448071] LustreError: 83027:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5636.545923] LustreError: 83058:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5639.559625] LustreError: lustre-MDT0000-mdc-ffff942411353800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5639.614492] LustreError: 83072:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff942411353800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff942407021d00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5639.661040] LustreError: 83072:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5640.624204] LustreError: 83058:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5647.930163] LustreError: lustre-MDT0000-mdc-ffff942411353800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5647.949524] LustreError: 83101:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff942411353800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff94240660b300) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5647.993385] LustreError: 83101:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5648.356834] LustreError: 83114:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5651.344557] LustreError: lustre-MDT0000-mdc-ffff942411353800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5651.434067] LustreError: 83133:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5651.445211] LustreError: 83133:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5652.440204] LustreError: 83114:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5652.451230] LustreError: 83114:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff942411353800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5652.593550] LustreError: 83140:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5652.620793] LustreError: 83140:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5655.011509] LustreError: 83162:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5655.023141] LustreError: 83162:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5658.886400] LustreError: 83189:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5661.826044] LustreError: lustre-MDT0000-mdc-ffff942411353800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5661.835216] LustreError: 83204:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff942411353800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff94240660be00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5662.968446] LustreError: 83189:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5670.110063] LustreError: lustre-MDT0000-mdc-ffff942411353800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5670.538625] LustreError: 83245:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5673.620501] LustreError: lustre-MDT0000-mdc-ffff942411353800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5673.647604] LustreError: 83260:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff942411353800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff942407021500) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5673.679863] LustreError: 83260:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5674.632344] LustreError: 83245:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5677.666400] LustreError: lustre-MDT0000-mdc-ffff942411353800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5677.744043] LustreError: 83293:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5677.759230] LustreError: 83293:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5678.841936] LustreError: 83273:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff942411353800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5686.794470] LustreError: 83345:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5686.816673] LustreError: 83345:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5690.603546] LustreError: lustre-MDT0000-mdc-ffff942411353800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5690.696758] LustreError: 83367:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5690.708771] LustreError: 83367:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5690.896223] LustreError: 83345:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5690.909893] LustreError: 83345:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5690.919238] LustreError: 83345:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff942411353800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5701.803245] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 23:47:59 (1787629679)
[ 5720.971566] LustreError: 79772:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5720.987500] LustreError: 79772:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5725.048608] LustreError: 79772:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5725.052119] LustreError: 79772:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5732.789685] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 23:48:30 (1787629710)
[ 5783.596754] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 23:49:20 (1787629760)
[ 5848.300297] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 23:50:25 (1787629825)
[ 5883.216836] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 23:51:00 (1787629860)
[ 5884.036391] Lustre: Mounted lustre-client - version 2.17.57_80_ga3b037e
[ 6007.881951] Lustre: Unmounted lustre-client
[ 6014.314301] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 23:53:11 (1787629991)
[ 6015.007214] LustreError: 87675:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 6017.373843] LustreError: lustre-OST0000-osc-ffff942411353800: operation ost_statfs to node 192.168.202.155@tcp failed: rc = -107
[ 6017.380746] LustreError: Skipped 9 previous similar messages
[ 6017.397462] LustreError: lustre-OST0000-osc-ffff942411353800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6017.412931] LustreError: 87762:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff942411353800: namespace resource [0x240000400:0x3f8d:0x0].0x0 (ffff94240660b100) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6017.469638] LustreError: 87762:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 6017.744138] LustreError: 87675:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout interrupted
[ 6017.751952] Lustre: 2358:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.155@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 6023.075403] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 23:53:21 (1787630001)
[ 6080.482093] Lustre: 88341:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787630004/real 1787630004]  req@ffff94241078ed80 x1874459701687680/t0(0) o101->lustre-MDT0000-mdc-ffff942411353800@192.168.202.155@tcp:12/10 lens 328/344 e 0 to 1 dl 1787630059 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 6080.577160] Lustre: 88341:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 24 previous similar messages
[ 6092.693172] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 23:54:29 (1787630069)
[ 6094.430861] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 6096.645476] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 23:54:34 (1787630074)
[ 6161.096161] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6163.210716] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 23:55:40 (1787630140)
[ 6221.971705] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 23:56:39 (1787630199)
[ 6224.383801] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6226.501942] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 23:56:43 (1787630203)
[ 6228.176482] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6231.038209] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 23:56:48 (1787630208)
[ 6272.859333] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 23:57:30 (1787630250)
[ 6285.456506] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6311.906047] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 192.168.202.155@tcp) was lost; in progress operations using this service will fail
[ 6311.926479] LustreError: Skipped 2 previous similar messages
[ 6318.048248] Lustre: lustre-MDT0000-mdc-ffff942411353800: Connection to lustre-MDT0000 (at 192.168.202.155@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6318.075538] Lustre: Skipped 16 previous similar messages
[ 6322.223385] Lustre: Evicted from MGS (at 192.168.202.155@tcp) after server handle changed from 0xf0a4e92a43f8d30 to 0xf0a4e92a43fa539
[ 6322.236945] Lustre: Skipped 2 previous similar messages
[ 6322.247624] Lustre: MGC192.168.202.155@tcp: Connection restored to 192.168.202.155@tcp (at 192.168.202.155@tcp)
[ 6322.259242] Lustre: Skipped 15 previous similar messages
[ 6323.504930] Lustre: 68269:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.155@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6340.153181] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6341.955685] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6353.085210] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 23:58:50 (1787630330)
[ 6356.144621] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6359.208521] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 23:58:55 (1787630335)
[ 6380.001787] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 192.168.202.155@tcp) was lost; in progress operations using this service will fail
[ 6390.454481] Lustre: Evicted from MGS (at 192.168.202.155@tcp) after server handle changed from 0xf0a4e92a43fa539 to 0xf0a4e92a43fabc2
[ 6399.650400] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 23:59:37 (1787630377)
[ 6459.567608] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 00:00:37 (1787630437)
[ 6539.524365] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6544.618505] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6632.416330] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 192.168.202.155@tcp) was lost; in progress operations using this service will fail
[ 6632.437105] LustreError: Skipped 1 previous similar message
[ 6642.683071] Lustre: Evicted from MGS (at 192.168.202.155@tcp) after server handle changed from 0xf0a4e92a43fac0f to 0xf0a4e92a43fba4e
[ 6642.691943] Lustre: Skipped 1 previous similar message
[ 6644.092813] Lustre: 68269:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.155@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6663.385942] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6665.657714] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6682.592372] Lustre: 2361:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787630600/real 1787630600]  req@ffff94243063ca80 x1874459708255104/t0(0) o400->lustre-MDT0000-mdc-ffff942411353800@192.168.202.155@tcp:12/10 lens 224/224 e 0 to 1 dl 1787630661 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6682.640331] Lustre: 2361:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 18 previous similar messages
[ 6700.230733] Lustre: 68269:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.155@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6720.159073] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6721.977730] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6866.659935] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 00:07:23 (1787630843)
[ 6879.310270] LustreError: lustre-OST0000-osc-ffff942411353800: operation ost_setattr to node 192.168.202.155@tcp failed: rc = -19
[ 6879.328320] LustreError: Skipped 348 previous similar messages
[ 6941.485534] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6945.482275] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 7216.041747] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 00:13:13 (1787631193)
[ 7354.346831] Lustre: lustre-MDT0000-mdc-ffff942411353800: Connection to lustre-MDT0000 (at 192.168.202.155@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7354.365379] Lustre: Skipped 6 previous similar messages
[ 7369.699732] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 192.168.202.155@tcp) was lost; in progress operations using this service will fail
[ 7369.710400] LustreError: Skipped 1 previous similar message
[ 7369.721624] Lustre: Evicted from MGS (at 192.168.202.155@tcp) after server handle changed from 0xf0a4e92a43fbf41 to 0xf0a4e92a44f6348
[ 7369.730166] Lustre: Skipped 1 previous similar message
[ 7369.770234] Lustre: MGC192.168.202.155@tcp: Connection restored to 192.168.202.155@tcp (at 192.168.202.155@tcp)
[ 7369.780740] Lustre: Skipped 9 previous similar messages
[ 7403.962777] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 00:16:21 (1787631381)
[ 7405.648734] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7407.555513] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 00:16:25 (1787631385)
[ 7409.989831] LustreError: lustre-MDT0000-mdc-ffff942411353800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7418.588508] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 00:16:36 (1787631396)
[ 7585.909733] LustreError: lustre-OST0000-osc-ffff942411353800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7587.744541] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 00:19:25 (1787631565)
[ 7612.896182] Lustre: 2358:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787631571/real 1787631571]  req@ffff94241dd89500 x1874459719460864/t0(0) o4->lustre-OST0000-osc-ffff942411353800@192.168.202.155@tcp:6/4 lens 4584/448 e 0 to 1 dl 1787631591 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7612.920165] Lustre: 2358:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 14 previous similar messages
[ 7630.509265] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 00:20:08 (1787631608)
[ 7631.789574] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7633.462202] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 00:20:11 (1787631611)
[ 7634.990905] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7637.051745] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 00:20:14 (1787631614)
[ 7674.562173] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 00:20:51 (1787631651)
[ 7714.295758] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 192.168.202.155@tcp) was lost; in progress operations using this service will fail
[ 7714.330874] Lustre: Evicted from MGS (at 192.168.202.155@tcp) after server handle changed from 0xf0a4e92a44f6348 to 0xf0a4e92a44f8b9d
[ 7714.348525] Lustre: Skipped 1 previous similar message
[ 7728.823810] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 00:21:46 (1787631706)
[ 7731.108871] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7733.307795] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 00:21:50 (1787631710)
[ 7735.128408] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7736.813552] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 00:21:54 (1787631714)
[ 7748.232812] Lustre: Unmounted lustre-client
[ 7750.305467] Lustre: Mounted lustre-client - version 2.17.57_80_ga3b037e
[ 7753.099707] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7783.290407] Lustre: 106829:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.155@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 7794.797260] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2