[ 1553.108307] LustreError: lustre-OST0000-osc-ffff948650f4b000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1553.127060] Lustre: 2351:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.102@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1557.754526] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 20:02:35 (1788739355)
[ 1559.980642] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1561.421094] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 20:02:39 (1788739359)
[ 1563.001409] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1564.249173] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 20:02:42 (1788739362)
[ 1584.620199] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 192.168.203.102@tcp) was lost; in progress operations using this service will fail
[ 1584.634680] Lustre: Evicted from MGS (at 192.168.203.102@tcp) after server handle changed from 0x88e3328bf5a3f20 to 0x88e3328bf5a5d73
[ 1685.711162] LustreError: lustre-MDT0000-mdc-ffff948650f4b000: operation ldlm_enqueue to node 192.168.203.102@tcp failed: rc = -19
[ 1685.736824] LustreError: Skipped 4 previous similar messages
[ 1703.394624] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 192.168.203.102@tcp) was lost; in progress operations using this service will fail
[ 1713.642959] Lustre: Evicted from MGS (at 192.168.203.102@tcp) after server handle changed from 0x88e3328bf5a5d73 to 0x88e3328bf5c4dc4
[ 1731.284828] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 20:05:29 (1788739529)
[ 1732.121665] Lustre: *** cfs_fail_loc=305, val=0***
[ 1732.125400] Lustre: Skipped 4 previous similar messages
[ 1774.047425] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 192.168.203.102@tcp) was lost; in progress operations using this service will fail
[ 1784.292699] Lustre: Evicted from MGS (at 192.168.203.102@tcp) after server handle changed from 0x88e3328bf5c4dc4 to 0x88e3328bf5c5366
[ 1801.019346] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 1803.273478] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1812.959621] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 20:06:50 (1788739610)
[ 1830.395355] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 192.168.203.102@tcp) was lost; in progress operations using this service will fail
[ 1830.431721] Lustre: Evicted from MGS (at 192.168.203.102@tcp) after server handle changed from 0x88e3328bf5c5366 to 0x88e3328bf5c5bd2
[ 1830.511848] LustreError: lustre-MDT0000-mdc-ffff948650f4b000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 1853.990429] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 20:07:31 (1788739651)
[ 1869.135673] LustreError: lustre-OST0000-osc-ffff948650f4b000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1888.324688] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 20:08:05 (1788739685)
[ 1903.553451] LustreError: lustre-MDT0000-mdc-ffff948650f4b000: operation mds_reint to node 192.168.203.102@tcp failed: rc = -19
[ 1903.562512] LustreError: Skipped 1 previous similar message
[ 1923.489579] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 192.168.203.102@tcp) was lost; in progress operations using this service will fail
[ 1933.741967] Lustre: Evicted from MGS (at 192.168.203.102@tcp) after server handle changed from 0x88e3328bf5c5bd2 to 0x88e3328bf5c9ce6
[ 1954.929415] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 1957.690441] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2026.953616] Lustre: lustre-MDT0000-mdc-ffff948650f4b000: Connection to lustre-MDT0000 (at 192.168.203.102@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2027.004390] Lustre: Skipped 13 previous similar messages
[ 2045.407162] Lustre: 2351:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788739828/real 1788739828]  req@ffff948648a63800 x1875627580175104/t0(0) o400->MGC192.168.203.102@tcp@192.168.203.102@tcp:26/25 lens 224/224 e 0 to 1 dl 1788739844 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2045.445672] Lustre: 2351:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 16 previous similar messages
[ 2045.459832] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 192.168.203.102@tcp) was lost; in progress operations using this service will fail
[ 2055.657098] Lustre: Evicted from MGS (at 192.168.203.102@tcp) after server handle changed from 0x88e3328bf5c9ce6 to 0x88e3328bf5dfc8a
[ 2055.670778] Lustre: MGC192.168.203.102@tcp: Connection restored to 192.168.203.102@tcp (at 192.168.203.102@tcp)
[ 2055.677759] Lustre: Skipped 18 previous similar messages
[ 2071.788115] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2073.908764] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2139.681838] LustreError: lustre-MDT0000-mdc-ffff948650f4b000: operation mds_reint to node 192.168.203.102@tcp failed: rc = -19
[ 2139.689647] LustreError: Skipped 2 previous similar messages
[ 2157.024081] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 192.168.203.102@tcp) was lost; in progress operations using this service will fail
[ 2167.286728] Lustre: Evicted from MGS (at 192.168.203.102@tcp) after server handle changed from 0x88e3328bf5dfc8a to 0x88e3328bf60037f
[ 2183.567640] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2185.544266] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2214.357318] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 20:13:32 (1788740012)
[ 2237.010968] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 192.168.203.102@tcp) was lost; in progress operations using this service will fail
[ 2237.033600] Lustre: Evicted from MGS (at 192.168.203.102@tcp) after server handle changed from 0x88e3328bf60037f to 0x88e3328bf60ce2b
[ 2243.800361] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2270.595780] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2301.037539] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2334.266233] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2375.648657] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 192.168.203.102@tcp) was lost; in progress operations using this service will fail
[ 2375.680710] LustreError: Skipped 3 previous similar messages
[ 2375.720967] Lustre: Evicted from MGS (at 192.168.203.102@tcp) after server handle changed from 0x88e3328bf61228d to 0x88e3328bf61ba64
[ 2375.729213] Lustre: Skipped 3 previous similar messages
[ 2385.624750] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2413.198572] LustreError: lustre-MDT0000-mdc-ffff948650f4b000: operation ldlm_enqueue to node 192.168.203.102@tcp failed: rc = -19
[ 2413.209036] LustreError: Skipped 8 previous similar messages
[ 2437.281797] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2516.652808] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 20:18:34 (1788740314)
[ 2571.101721] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 2573.038380] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 2861.064867] Lustre: lustre-OST0000-osc-ffff948650f4b000: Connection to lustre-OST0000 (at 192.168.203.102@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2861.076543] Lustre: Skipped 7 previous similar messages
[ 2879.088132] Lustre: lustre-OST0000-osc-ffff948650f4b000: Connection restored to 192.168.203.102@tcp (at 192.168.203.102@tcp)
[ 2879.093060] Lustre: Skipped 16 previous similar messages
[ 2893.635541] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 2896.061977] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3194.612313] LustreError: lustre-OST0000-osc-ffff948650f4b000: operation ldlm_enqueue to node 192.168.203.102@tcp failed: rc = -19
[ 3194.630080] LustreError: Skipped 6 previous similar messages
[ 3247.286677] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3250.688251] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3490.884936] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 20:34:48 (1788741288)
[ 3508.703960] Lustre: 50520:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788741291/real 1788741291]  req@ffff948650fc2a00 x1875627605034496/t0(0) o101->lustre-MDT0000-mdc-ffff948650f4b000@192.168.203.102@tcp:12/10 lens 576/1152 e 0 to 1 dl 1788741307 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3508.747888] Lustre: 50520:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
[ 3508.756236] Lustre: lustre-MDT0000-mdc-ffff948650f4b000: Connection to lustre-MDT0000 (at 192.168.203.102@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3508.779147] Lustre: Skipped 1 previous similar message
[ 3508.805377] Lustre: lustre-MDT0000-mdc-ffff948650f4b000: Connection restored to 192.168.203.102@tcp (at 192.168.203.102@tcp)
[ 3508.814954] Lustre: Skipped 1 previous similar message
[ 3515.674548] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 20:35:13 (1788741313)
[ 3541.729902] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 20:35:39 (1788741339)
[ 3567.512216] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 20:36:05 (1788741365)
[ 3568.148047] Lustre: Mounted lustre-client - version 2.17.58_3_g671df26
[ 3598.865557] Lustre: 2353:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788741382/real 1788741382]  req@ffff948641497480 x1875627605053312/t0(0) o400->MGC192.168.203.102@tcp@192.168.203.102@tcp:26/25 lens 224/224 e 0 to 1 dl 1788741398 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3598.939960] Lustre: 2353:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 3598.955165] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 192.168.203.102@tcp) was lost; in progress operations using this service will fail
[ 3598.979679] LustreError: Skipped 2 previous similar messages
[ 3609.586064] Lustre: Evicted from MGS (at 192.168.203.102@tcp) after server handle changed from 0x88e3328bf62fa5e to 0x88e3328bf7b32c8
[ 3609.598587] Lustre: Skipped 2 previous similar messages
[ 3611.384824] Lustre: 15913:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.102@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 3628.931257] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 3630.568196] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3633.025772] Lustre: Unmounted lustre-client
[ 3640.105757] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 20:37:17 (1788741437)
[ 3748.896637] Lustre: 2350:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788741532/real 1788741532]  req@ffff948650fc1500 x1875627605089792/t0(0) o4->lustre-OST0000-osc-ffff948650f4b000@192.168.203.102@tcp:6/4 lens 488/448 e 0 to 1 dl 1788741548 ref 2 fl Rpc:XQr/602/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 3748.929363] Lustre: 2350:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 42 previous similar messages
[ 3940.773552] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 20:42:18 (1788741738)
[ 3989.082524] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 20:43:06 (1788741786)
[ 3992.340928] Lustre: Unmounted lustre-client
[ 4013.739462] Lustre: Mounted lustre-client - version 2.17.58_3_g671df26
[ 4020.406791] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 20:43:38 (1788741818)
[ 4020.772936] LustreError: 56447:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4021.832105] LustreError: 56447:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4022.034189] Lustre: *** cfs_fail_loc=305, val=0***
[ 4045.676612] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 20:44:03 (1788741843)
[ 4046.170327] Lustre: Mounted lustre-client - version 2.17.58_3_g671df26
[ 4048.969279] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4059.263242] Lustre: Unmounted lustre-client
[ 4065.259830] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 20:44:23 (1788741863)
[ 4142.557433] LustreError: lustre-MDT0000-mdc-ffff94864731e800: operation mds_reint to node 192.168.203.102@tcp failed: rc = -19
[ 4142.564609] Lustre: lustre-MDT0000-mdc-ffff94864731e800: Connection to lustre-MDT0000 (at 192.168.203.102@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4142.575547] Lustre: Skipped 23 previous similar messages
[ 4158.954059] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 192.168.203.102@tcp) was lost; in progress operations using this service will fail
[ 4158.983554] Lustre: Evicted from MGS (at 192.168.203.102@tcp) after server handle changed from 0x88e3328bf7b502d to 0x88e3328bf7e0b1c
[ 4158.992901] Lustre: MGC192.168.203.102@tcp: Connection restored to 192.168.203.102@tcp (at 192.168.203.102@tcp)
[ 4158.997745] Lustre: Skipped 24 previous similar messages
[ 4279.200795] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 20:47:56 (1788742076)
[ 4283.934562] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4297.203588] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 192.168.203.102@tcp) was lost; in progress operations using this service will fail
[ 4297.229741] Lustre: Evicted from MGS (at 192.168.203.102@tcp) after server handle changed from 0x88e3328bf7e0b1c to 0x88e3328bf82c81b
[ 4297.320952] LustreError: lustre-MDT0000-mdc-ffff94864731e800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4312.619289] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 20:48:30 (1788742110)
[ 4312.998789] Lustre: Mounted lustre-client - version 2.17.58_3_g671df26
[ 4320.140706] LustreError: lustre-OST0000-osc-ffff94864731e800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4330.463347] Lustre: 2353:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788742113/real 1788742113]  req@ffff948645d96300 x1875627609736960/t0(0) o103->lustre-OST0000-osc-ffff9486591f1000@192.168.203.102@tcp:17/18 lens 328/224 e 0 to 1 dl 1788742129 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295
[ 4330.499434] Lustre: 2353:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 113 previous similar messages
[ 4333.855671] Lustre: Unmounted lustre-client
[ 4341.435423] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 20:48:58 (1788742138)
[ 4348.009105] LustreError: lustre-MDT0000-mdc-ffff94864731e800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4348.026175] LustreError: 60914:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff94864731e800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff9486492aae00) refcount nonzero (2) after lock cleanup; forcing cleanup.
[ 4348.053847] LustreError: 60830:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4354.147624] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 20:49:12 (1788742152)
[ 4354.354032] LustreError: 61531:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4359.647149] LustreError: 61531:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=0
[ 4359.651740] LustreError: 61531:0:(import.c:755:ptlrpc_connect_import_locked()) already connected
[ 4360.204409] LustreError: lustre-MDT0000-mdc-ffff94864731e800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4360.233713] LustreError: 61552:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4360.343878] LustreError: 61557:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4361.456927] LustreError: 61563:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4361.472399] LustreError: 61563:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4363.716869] LustreError: 61586:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4363.722142] LustreError: 61586:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4368.155113] LustreError: 61631:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4368.168123] LustreError: 61631:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4380.532473] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 20:49:38 (1788742178)
[ 4420.347692] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4422.028966] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4434.468262] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 20:50:31 (1788742231)
[ 4476.523457] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4478.018807] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4489.112558] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 20:51:25 (1788742285)
[ 4553.722646] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4555.142525] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4564.090975] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 20:52:41 (1788742361)
[ 4602.699823] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4604.468179] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4610.244566] Lustre: Unmounted lustre-client
[ 4662.805918] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4665.134896] Lustre: Mounted lustre-client - version 2.17.58_3_g671df26
[ 4672.301720] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 20:54:29 (1788742469)
[ 4675.246827] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 4677.336357] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 20:54:34 (1788742474)
[ 4710.370994] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 20:55:08 (1788742508)
[ 4711.853883] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 4713.372375] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 20:55:11 (1788742511)
[ 4714.652852] Lustre: *** cfs_fail_loc=805, val=0***
[ 4714.687671] Lustre: Mounted lustre-client - version 2.17.58_3_g671df26
[ 4719.214329] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4740.590210] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 192.168.203.102@tcp) was lost; in progress operations using this service will fail
[ 4740.628644] Lustre: Evicted from MGS (at 192.168.203.102@tcp) after server handle changed from 0x88e3328bf82d239 to 0x88e3328bf82d5d5
[ 4744.722775] Lustre: Unmounted lustre-client
[ 4752.035920] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 20:55:49 (1788742549)
[ 4755.943150] Lustre: lustre-MDT0000-mdc-ffff948648169800: Connection to lustre-MDT0000 (at 192.168.203.102@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4755.956144] Lustre: Skipped 14 previous similar messages
[ 4776.453111] Lustre: MGC192.168.203.102@tcp: Connection restored to 192.168.203.102@tcp (at 192.168.203.102@tcp)
[ 4776.464507] Lustre: Skipped 15 previous similar messages
[ 4776.468870] LustreError: 2349:0:(client.c:3449:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff948645d97b80 x1875627609818624/t85899345924(85899345924) o101->lustre-MDT0000-mdc-ffff948648169800@192.168.203.102@tcp:12/10 lens 648/608 e 0 to 0 dl 1788742591 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 4784.383702] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4785.465543] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4791.424822] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 20:56:29 (1788742589)
[ 4792.455436] LustreError: lustre-OST0000-osc-ffff948648169800: operation ost_write to node 192.168.203.102@tcp failed: rc = -107
[ 4792.465832] LustreError: Skipped 3 previous similar messages
[ 4792.477706] LustreError: lustre-OST0000-osc-ffff948648169800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4792.498813] Lustre: 2350:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.102@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 4792.523536] LustreError: 72739:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff948648169800: namespace resource [0x240000400:0x4402:0x0].0x0 (ffff94864400c600) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4798.027536] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 20:56:36 (1788742596)
[ 4799.003873] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 4800.162318] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 20:56:38 (1788742598)
[ 4801.214401] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 4802.309463] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 20:56:40 (1788742600)
[ 4803.273995] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 4804.424256] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 20:56:42 (1788742602)
[ 4805.308537] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 4806.282751] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 20:56:44 (1788742604)
[ 4807.518577] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 4808.667728] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 20:56:46 (1788742606)
[ 4809.821970] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 4811.060332] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 20:56:49 (1788742609)
[ 4812.235587] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 4813.411980] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 20:56:51 (1788742611)
[ 4814.409201] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 4815.579922] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 20:56:53 (1788742613)
[ 4816.671718] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 4817.845827] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 20:56:56 (1788742616)
[ 4818.764647] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 4819.986993] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 20:56:58 (1788742618)
[ 4821.090654] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 4822.494769] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 20:57:00 (1788742620)
[ 4824.459014] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 4825.860956] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 20:57:03 (1788742623)
[ 4852.167552] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 20:57:30 (1788742650)
[ 4879.960426] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 20:57:57 (1788742677)
[ 4880.370108] Lustre: *** cfs_fail_loc=51b, val=3***
[ 4889.179241] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 20:58:07 (1788742687)
[ 4889.478036] Lustre: *** cfs_fail_loc=51b, val=4***
[ 4897.603880] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 20:58:15 (1788742695)
[ 4897.851432] Lustre: *** cfs_fail_loc=50f, val=3***
[ 4903.641968] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 20:58:21 (1788742701)
[ 4903.790218] Lustre: *** cfs_fail_loc=50f, val=4***
[ 4910.481880] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 20:58:28 (1788742708)
[ 4910.687245] Lustre: *** cfs_fail_loc=510, val=3***
[ 4917.964120] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 20:58:35 (1788742715)
[ 4918.291138] Lustre: *** cfs_fail_loc=51b, val=3***
[ 4977.119164] Lustre: 2351:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788742721/real 1788742721]  req@ffff9486441ad500 x1875627609857792/t0(0) o3->lustre-OST0000-osc-ffff948648169800@192.168.203.102@tcp:6/4 lens 488/4536 e 0 to 1 dl 1788742776 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4977.149386] Lustre: 2351:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
[ 4984.932981] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 20:59:42 (1788742782)
[ 4985.671925] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5050.658300] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 21:00:48 (1788742848)
[ 5051.027564] LustreError: 82948:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5053.624265] LustreError: lustre-MDT0000-mdc-ffff948648169800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5053.645456] LustreError: 82962:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff948648169800: inode [0x20000a042:0xe:0x0] mdc close failed: rc = -108
[ 5053.682731] LustreError: 82962:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff948648169800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff948645f2b900) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5053.710825] LustreError: 82962:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5053.716693] LustreError: 82967:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5053.737589] LustreError: 82967:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5055.087612] LustreError: 82948:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5057.199694] LustreError: 82978:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5060.060866] LustreError: lustre-MDT0000-mdc-ffff948648169800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5060.071611] LustreError: 82992:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff948648169800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff948645ea3c00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5060.078283] LustreError: 82992:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5061.279464] LustreError: 82978:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5068.298068] LustreError: lustre-MDT0000-mdc-ffff948648169800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5068.717519] LustreError: 83045:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5071.484333] LustreError: lustre-MDT0000-mdc-ffff948648169800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5071.542568] LustreError: 83064:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5072.799240] LustreError: 83045:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5072.804875] LustreError: 83045:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff948648169800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5073.733371] LustreError: 83082:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5073.744690] LustreError: 83082:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 12 previous similar messages
[ 5078.412458] LustreError: 83120:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5081.009254] LustreError: lustre-MDT0000-mdc-ffff948648169800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5081.026555] LustreError: 83134:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff948648169800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff94864400ca00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5081.039049] LustreError: 83134:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5082.487133] LustreError: 83120:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5089.003414] LustreError: lustre-MDT0000-mdc-ffff948648169800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5089.331139] LustreError: 83176:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5092.137868] LustreError: lustre-MDT0000-mdc-ffff948648169800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5092.157465] LustreError: 83190:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff948648169800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff948645f2bb00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5092.167713] LustreError: 83190:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5093.407190] LustreError: 83176:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5096.321568] LustreError: lustre-MDT0000-mdc-ffff948648169800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5096.405151] LustreError: 83223:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5096.419867] LustreError: 83223:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5097.560746] LustreError: 83204:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff948648169800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5105.218586] LustreError: 83275:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5105.222572] LustreError: 83275:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5108.457063] LustreError: lustre-MDT0000-mdc-ffff948648169800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5108.579803] LustreError: 83298:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5108.586650] LustreError: 83298:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5109.303875] LustreError: 83275:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5109.314147] LustreError: 83275:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5109.323471] LustreError: 83275:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff948648169800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5119.426112] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 21:01:57 (1788742917)
[ 5139.004670] LustreError: 68197:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5139.012272] LustreError: 68197:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5143.079138] LustreError: 68197:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5143.086744] LustreError: 68197:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5149.964523] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 21:02:27 (1788742947)
[ 5199.664282] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 21:03:17 (1788742997)
[ 5260.959723] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 21:04:19 (1788743059)
[ 5295.476753] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 21:04:53 (1788743093)
[ 5296.051735] Lustre: Mounted lustre-client - version 2.17.58_3_g671df26
[ 5420.369613] Lustre: Unmounted lustre-client
[ 5431.233889] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 21:07:08 (1788743228)
[ 5432.115314] LustreError: 87605:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5434.768515] LustreError: lustre-OST0000-osc-ffff948648169800: operation ldlm_enqueue to node 192.168.203.102@tcp failed: rc = -107
[ 5434.779713] LustreError: Skipped 9 previous similar messages
[ 5434.789401] Lustre: lustre-OST0000-osc-ffff948648169800: Connection to lustre-OST0000 (at 192.168.203.102@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5434.805825] Lustre: Skipped 16 previous similar messages
[ 5434.814930] LustreError: lustre-OST0000-osc-ffff948648169800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5434.824315] LustreError: 87688:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff948648169800: namespace resource [0x240000400:0x442d:0x0].0x0 (ffff948645ea3700) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5434.834420] LustreError: 87688:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5435.223919] LustreError: 87605:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5435.235250] Lustre: 2352:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.102@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5441.190166] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 21:07:19 (1788743239)
[ 5498.869205] Lustre: lustre-MDT0000-mdc-ffff948648169800: Connection restored to 192.168.203.102@tcp (at 192.168.203.102@tcp)
[ 5498.876893] Lustre: Skipped 15 previous similar messages
[ 5503.942257] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 21:08:22 (1788743302)
[ 5505.018724] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 5506.389479] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 21:08:24 (1788743304)
[ 5569.798368] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 5571.904431] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 21:09:29 (1788743369)
[ 5627.871213] Lustre: 90005:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788743372/real 1788743372]  req@ffff9486441ae300 x1875627610012928/t0(0) o36->lustre-MDT0000-mdc-ffff948648169800@192.168.203.102@tcp:12/10 lens 496/440 e 0 to 1 dl 1788743427 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ln.0' uid:0 gid:0 projid:4294967295
[ 5627.910729] Lustre: 90005:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
[ 5629.949836] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 21:10:27 (1788743427)
[ 5633.205718] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 5635.369532] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 21:10:32 (1788743432)
[ 5637.172601] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 5638.866769] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 21:10:36 (1788743436)
[ 5670.972446] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 21:11:08 (1788743468)
[ 5682.580947] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5709.799536] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 192.168.203.102@tcp) was lost; in progress operations using this service will fail
[ 5709.818615] LustreError: Skipped 2 previous similar messages
[ 5709.846578] Lustre: Evicted from MGS (at 192.168.203.102@tcp) after server handle changed from 0x88e3328bf82e071 to 0x88e3328bf82f8dc
[ 5709.857952] Lustre: Skipped 2 previous similar messages
[ 5711.572563] Lustre: 68200:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.102@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 5733.349462] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5734.874700] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5744.592335] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 21:12:22 (1788743542)
[ 5746.730919] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 5748.644407] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 21:12:26 (1788743546)
[ 5761.021095] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 192.168.203.102@tcp) was lost; in progress operations using this service will fail
[ 5761.044623] Lustre: Evicted from MGS (at 192.168.203.102@tcp) after server handle changed from 0x88e3328bf82f8dc to 0x88e3328bf82fd0b
[ 5775.361207] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 21:12:53 (1788743573)
[ 5827.280473] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 21:13:44 (1788743624)
[ 5882.235600] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5885.321946] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5968.351236] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 192.168.203.102@tcp) was lost; in progress operations using this service will fail
[ 5968.362123] LustreError: Skipped 1 previous similar message
[ 5978.793508] Lustre: Evicted from MGS (at 192.168.203.102@tcp) after server handle changed from 0x88e3328bf8300a0 to 0x88e3328bf8310d7
[ 5978.804755] Lustre: Skipped 1 previous similar message
[ 5985.550480] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5987.531268] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6025.088684] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6026.838976] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6156.144175] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 21:19:13 (1788743953)
[ 6168.556810] Lustre: lustre-OST0000-osc-ffff948648169800: Connection to lustre-OST0000 (at 192.168.203.102@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6168.571184] Lustre: Skipped 9 previous similar messages
[ 6211.743618] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6214.929255] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6469.645329] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 21:24:27 (1788744267)
[ 6603.753411] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 192.168.203.102@tcp) was lost; in progress operations using this service will fail
[ 6603.771115] LustreError: Skipped 1 previous similar message
[ 6603.803813] Lustre: Evicted from MGS (at 192.168.203.102@tcp) after server handle changed from 0x88e3328bf831219 to 0x88e3328bf92b491
[ 6603.810340] Lustre: Skipped 1 previous similar message
[ 6603.820470] Lustre: MGC192.168.203.102@tcp: Connection restored to 192.168.203.102@tcp (at 192.168.203.102@tcp)
[ 6603.830976] Lustre: Skipped 12 previous similar messages
[ 6640.576324] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 21:27:18 (1788744438)
[ 6641.845807] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 6643.407406] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 21:27:21 (1788744441)
[ 6645.207210] LustreError: lustre-MDT0000-mdc-ffff948648169800: operation ldlm_enqueue to node 192.168.203.102@tcp failed: rc = -107
[ 6645.217713] LustreError: Skipped 24 previous similar messages
[ 6645.228719] LustreError: lustre-MDT0000-mdc-ffff948648169800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6651.818162] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 21:27:29 (1788744449)
[ 6820.839382] LustreError: lustre-OST0000-osc-ffff948648169800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6822.776978] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 21:30:20 (1788744620)
[ 6848.479634] Lustre: 2350:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788744627/real 1788744627]  req@ffff948648e2bb80 x1875627627696640/t0(0) o4->lustre-OST0000-osc-ffff948648169800@192.168.203.102@tcp:6/4 lens 4584/448 e 0 to 1 dl 1788744647 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 6848.503571] Lustre: 2350:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 24 previous similar messages
[ 6848.515132] Lustre: lustre-OST0000-osc-ffff948648169800: Connection to lustre-OST0000 (at 192.168.203.102@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6848.520619] Lustre: Skipped 3 previous similar messages
[ 6867.824464] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 21:31:05 (1788744665)
[ 6869.834067] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 6871.268406] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 21:31:09 (1788744669)
[ 6872.915450] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 6874.690301] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 21:31:12 (1788744672)
[ 6908.657845] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 21:31:46 (1788744706)
[ 6945.762786] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 192.168.203.102@tcp) was lost; in progress operations using this service will fail
[ 6945.790227] Lustre: Evicted from MGS (at 192.168.203.102@tcp) after server handle changed from 0x88e3328bf92b491 to 0x88e3328bf92e082
[ 6945.805068] Lustre: Skipped 1 previous similar message
[ 6961.102929] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 21:32:38 (1788744758)
[ 6962.799062] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 6964.162326] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 21:32:42 (1788744762)
[ 6965.799667] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 6967.944843] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 21:32:45 (1788744765)
[ 6977.768068] Lustre: Unmounted lustre-client
[ 6980.094295] Lustre: Mounted lustre-client - version 2.17.58_3_g671df26
[ 6982.680621] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7001.567132] Lustre: 2351:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788744784/real 1788744784]  req@ffff948647408a80 x1875627627883008/t0(0) o400->MGC192.168.203.102@tcp@192.168.203.102@tcp:26/25 lens 224/224 e 0 to 1 dl 1788744800 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7022.265409] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2