[ 1471.907938] LustreError: lustre-OST0000-osc-ffff8800aa453800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1471.919499] Lustre: 1963:0:(llite_lib.c:4127:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.104@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1475.187474] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 02:01:46 (1734678106)
[ 1476.056749] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1476.832961] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 02:01:47 (1734678107)
[ 1477.647670] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1478.479794] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 02:01:49 (1734678109)
[ 1497.715610] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 192.168.203.104@tcp) was lost; in progress operations using this service will fail
[ 1497.765922] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0xd1ece635b81c68d3 to 0xd1ece635b81c8bf6
[ 1590.404813] LustreError: lustre-MDT0000-mdc-ffff8800aa453800: operation mds_close to node 192.168.203.104@tcp failed: rc = -19
[ 1590.413918] LustreError: Skipped 3 previous similar messages
[ 1607.912269] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 192.168.203.104@tcp) was lost; in progress operations using this service will fail
[ 1607.940739] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0xd1ece635b81c8bf6 to 0xd1ece635b82171b3
[ 1611.202451] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 02:04:02 (1734678242)
[ 1611.408396] Lustre: *** cfs_fail_loc=305, val=0***
[ 1611.410286] Lustre: Skipped 4 previous similar messages
[ 1648.011668] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 192.168.203.104@tcp) was lost; in progress operations using this service will fail
[ 1648.026086] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0xd1ece635b82171b3 to 0xd1ece635b8217708
[ 1648.763330] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1649.497017] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1653.567138] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 02:04:44 (1734678284)
[ 1661.661080] LustreError: lustre-MDT0000-mdc-ffff8800aa453800: operation mds_statfs to node 192.168.203.104@tcp failed: rc = -107
[ 1661.664207] LustreError: Skipped 1 previous similar message
[ 1661.690024] LustreError: lustre-MDT0000-mdc-ffff8800aa453800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 1663.029244] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 192.168.203.104@tcp) was lost; in progress operations using this service will fail
[ 1663.045465] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0xd1ece635b8217708 to 0xd1ece635b8217aa4
[ 1667.592355] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 02:04:58 (1734678298)
[ 1673.596646] LustreError: lustre-OST0000-osc-ffff8800aa453800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1681.290836] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 02:05:12 (1734678312)
[ 1708.101499] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 192.168.203.104@tcp) was lost; in progress operations using this service will fail
[ 1708.121966] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0xd1ece635b8217aa4 to 0xd1ece635b8223e34
[ 1714.378020] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1715.029130] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1751.091223] hrtimer: interrupt took 7227222 ns
[ 1793.346106] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 192.168.203.104@tcp) was lost; in progress operations using this service will fail
[ 1793.355848] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0xd1ece635b8223e34 to 0xd1ece635b826026d
[ 1800.043654] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1800.819173] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1863.148316] LustreError: lustre-MDT0000-mdc-ffff8800aa453800: operation mds_close to node 192.168.203.104@tcp failed: rc = -19
[ 1863.152892] LustreError: Skipped 7 previous similar messages
[ 1878.494164] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 192.168.203.104@tcp) was lost; in progress operations using this service will fail
[ 1878.501504] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0xd1ece635b826026d to 0xd1ece635b829927a
[ 1884.984435] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1885.736059] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1910.411864] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 02:09:01 (1734678541)
[ 1930.947970] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 1948.622313] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 192.168.203.104@tcp) was lost; in progress operations using this service will fail
[ 1948.629472] LustreError: Skipped 1 previous similar message
[ 1948.659496] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0xd1ece635b82b123f to 0xd1ece635b82b13ff
[ 1948.663689] Lustre: Skipped 1 previous similar message
[ 1950.209606] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 1956.384196] Lustre: lustre-MDT0000-mdc-ffff8800aa453800: Connection to lustre-MDT0000 (at 192.168.203.104@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 1956.391466] Lustre: Skipped 16 previous similar messages
[ 1973.688183] Lustre: MGC192.168.203.104@tcp: Connection restored to 192.168.203.104@tcp (at 192.168.203.104@tcp)
[ 1973.691544] Lustre: Skipped 25 previous similar messages
[ 1974.065131] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2002.467351] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2040.562530] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2083.768533] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2083.791804] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 192.168.203.104@tcp) was lost; in progress operations using this service will fail
[ 2083.805101] LustreError: Skipped 3 previous similar messages
[ 2083.819509] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0xd1ece635b82d2607 to 0xd1ece635b82e70ab
[ 2083.825002] Lustre: Skipped 3 previous similar messages
[ 2153.385524] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 02:13:04 (1734678784)
[ 2164.931127] LustreError: lustre-OST0000-osc-ffff8800aa453800: operation ldlm_enqueue to node 192.168.203.104@tcp failed: rc = -19
[ 2164.941688] LustreError: Skipped 10 previous similar messages
[ 2186.180837] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2187.254186] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 2518.081527] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2519.116867] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 2826.656731] LustreError: lustre-OST0000-osc-ffff8800aa453800: operation ldlm_enqueue to node 192.168.203.104@tcp failed: rc = -107
[ 2826.663050] LustreError: Skipped 1 previous similar message
[ 2826.665388] Lustre: lustre-OST0000-osc-ffff8800aa453800: Connection to lustre-OST0000 (at 192.168.203.104@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2826.676275] Lustre: Skipped 6 previous similar messages
[ 2846.879749] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2847.756493] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3119.226713] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 02:29:10 (1734679750)
[ 3135.755117] Lustre: 22332:0:(client.c:2364:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1734679751/real 1734679751]  req@ffff8800a85ed880 x1818940379372160/t0(0) o101->lustre-MDT0000-mdc-ffff8800aa453800@192.168.203.104@tcp:12/10 lens 576/1024 e 0 to 1 dl 1734679767 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0
[ 3135.770127] Lustre: 22332:0:(client.c:2364:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
[ 3135.784852] Lustre: lustre-MDT0000-mdc-ffff8800aa453800: Connection restored to 192.168.203.104@tcp (at 192.168.203.104@tcp)
[ 3135.788871] Lustre: Skipped 10 previous similar messages
[ 3138.451532] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 02:29:29 (1734679769)
[ 3158.201758] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 02:29:49 (1734679789)
[ 3178.441443] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 02:30:09 (1734679809)
[ 3178.654324] Lustre: Mounted lustre-client
[ 3203.701140] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 192.168.203.104@tcp) was lost; in progress operations using this service will fail
[ 3203.712234] LustreError: Skipped 1 previous similar message
[ 3203.718321] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0xd1ece635b830441c to 0xd1ece635b86079cb
[ 3203.724324] Lustre: Skipped 1 previous similar message
[ 3208.025166] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3208.726753] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3210.227261] Lustre: Unmounted lustre-client
[ 3212.697388] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 02:30:43 (1734679843)
[ 3218.687348] Lustre: 1965:0:(client.c:2364:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1734679830/real 1734679830]  req@ffff88012e39bb80 x1818940379394688/t0(0) o400->lustre-MDT0000-mdc-ffff8800aa453800@192.168.203.104@tcp:12/10 lens 224/224 e 0 to 1 dl 1734679846 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0
[ 3218.699809] Lustre: 1965:0:(client.c:2364:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
[ 3378.398402] Lustre: 1963:0:(client.c:2364:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1734679994/real 1734679994]  req@ffff8800a80e6d80 x1818940379453952/t0(0) o4->lustre-OST0000-osc-ffff8800aa453800@192.168.203.104@tcp:6/4 lens 488/448 e 0 to 1 dl 1734680010 ref 2 fl Rpc:XQr/202/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0
[ 3378.418597] Lustre: 1963:0:(client.c:2364:ptlrpc_expire_one_request()) Skipped 91 previous similar messages
[ 3442.470608] Lustre: lustre-OST0000-osc-ffff8800aa453800: Connection to lustre-OST0000 (at 192.168.203.104@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3442.480615] Lustre: Skipped 18 previous similar messages
[ 3498.794920] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 02:35:30 (1734680130)
[ 3542.480778] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 02:36:13 (1734680173)
[ 3543.994917] Lustre: Unmounted lustre-client
[ 3554.409526] Lustre: Mounted lustre-client
[ 3557.066071] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 02:36:28 (1734680188)
[ 3557.281829] LustreError: 28933:0:(mdc_locks.c:1330:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 3558.290276] LustreError: 28933:0:(mdc_locks.c:1330:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 3558.362561] Lustre: *** cfs_fail_loc=305, val=0***
[ 3576.337540] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 02:36:47 (1734680207)
[ 3576.463787] Lustre: Mounted lustre-client
[ 3576.829311] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 3586.876342] Lustre: Unmounted lustre-client
[ 3589.316973] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 02:37:00 (1734680220)
[ 3636.256174] LustreError: lustre-MDT0000-mdc-ffff8800a84b1800: operation mds_reint to node 192.168.203.104@tcp failed: rc = -19
[ 3652.553036] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 192.168.203.104@tcp) was lost; in progress operations using this service will fail
[ 3652.566468] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0xd1ece635b86089c3 to 0xd1ece635b86350c8
[ 3725.856719] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 02:39:16 (1734680356)
[ 3728.672308] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 3737.704247] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 192.168.203.104@tcp) was lost; in progress operations using this service will fail
[ 3737.729431] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0xd1ece635b86350c8 to 0xd1ece635b867fb75
[ 3737.735726] Lustre: MGC192.168.203.104@tcp: Connection restored to  (at 192.168.203.104@tcp)
[ 3737.740166] Lustre: Skipped 25 previous similar messages
[ 3737.757080] LustreError: lustre-MDT0000-mdc-ffff8800a84b1800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 3743.800411] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 02:39:35 (1734680375)
[ 3744.006659] Lustre: Mounted lustre-client
[ 3749.019978] LustreError: lustre-OST0000-osc-ffff8800a84b1800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 3761.022100] Lustre: 1966:0:(client.c:2364:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1734680376/real 1734680376]  req@ffff8800a8761f80 x1818940384097280/t0(0) o103->lustre-OST0000-osc-ffff8800aae61800@192.168.203.104@tcp:17/18 lens 328/224 e 0 to 1 dl 1734680392 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0
[ 3761.034273] Lustre: 1966:0:(client.c:2364:ptlrpc_expire_one_request()) Skipped 87 previous similar messages
[ 3761.668571] Lustre: Unmounted lustre-client
[ 3764.673035] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 02:39:55 (1734680395)
[ 3769.792763] LustreError: lustre-MDT0000-mdc-ffff8800a84b1800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 3769.800594] LustreError: 1318:0:(file.c:5963:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 3772.950410] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 02:40:04 (1734680404)
[ 3773.021206] LustreError: 2074:0:(recover.c:338:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 3777.514410] LustreError: lustre-MDT0000-mdc-ffff8800a84b1800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 3777.520505] LustreError: 2099:0:(import.c:307:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 3777.523732] LustreError: 2074:0:(recover.c:338:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=501
[ 3777.530790] LustreError: 2074:0:(import.c:719:ptlrpc_connect_import_locked()) already connecting
[ 3777.551954] LustreError: 2104:0:(file.c:5963:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3778.605643] LustreError: 2111:0:(file.c:5963:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3778.613485] LustreError: 2111:0:(file.c:5963:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 3780.726951] LustreError: 2135:0:(file.c:5963:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3780.731641] LustreError: 2135:0:(file.c:5963:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 3784.951929] LustreError: 2182:0:(file.c:5963:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3784.964807] LustreError: 2182:0:(file.c:5963:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 3791.147282] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 02:40:22 (1734680422)
[ 3814.490576] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3815.415452] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3821.521374] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 02:40:52 (1734680452)
[ 3844.915262] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3845.869418] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3851.387803] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 02:41:22 (1734680482)
[ 3898.582779] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3899.407208] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3904.360867] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 02:42:15 (1734680535)
[ 3925.793679] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3926.563553] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3929.575818] Lustre: Unmounted lustre-client
[ 3956.935860] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3958.669890] Lustre: Mounted lustre-client
[ 3961.873834] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 02:43:12 (1734680592)
[ 3963.285858] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 3964.133674] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 02:43:15 (1734680595)
[ 3978.583332] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 02:43:29 (1734680609)
[ 3979.267945] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 3980.153457] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 02:43:31 (1734680611)
[ 3981.025815] Lustre: *** cfs_fail_loc=805, val=0***
[ 3981.059122] Lustre: Mounted lustre-client
[ 3984.031542] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4000.904394] LustreError: 1962:0:(ldlm_resource.c:1149:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8800a84b2800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff880136b4fd00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4001.107630] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 192.168.203.104@tcp) was lost; in progress operations using this service will fail
[ 4001.124284] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0xd1ece635b868057e to 0xd1ece635b8680abe
[ 4001.477324] Lustre: Unmounted lustre-client
[ 4004.829932] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 02:43:55 (1734680635)
[ 4021.187133] LustreError: 1962:0:(client.c:3286:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff8800a80e6d80 x1818940384164992/t90194313219(90194313219) o101->lustre-MDT0000-mdc-ffff8800a8fb4000@192.168.203.104@tcp:12/10 lens 664/608 e 0 to 0 dl 1734680668 ref 2 fl Interpret:RPQU/204/0 rc 301/301 job:'multiop.0' uid:0 gid:0
[ 4026.868327] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4027.675906] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4032.891331] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 02:44:23 (1734680663)
[ 4033.853351] LustreError: lustre-OST0000-osc-ffff8800a8fb4000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4033.867882] Lustre: 1966:0:(llite_lib.c:4127:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.104@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 4033.901692] LustreError: 13928:0:(ldlm_resource.c:1149:ldlm_resource_complain()) lustre-OST0000-osc-ffff8800a8fb4000: namespace resource [0x240000400:0x7ee2:0x0].0x0 (ffff8800b3cd4e00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4033.908128] LustreError: 13928:0:(ldlm_resource.c:1149:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4039.332051] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 02:44:30 (1734680670)
[ 4040.146683] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 4041.170340] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 02:44:32 (1734680672)
[ 4042.022428] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 4043.080664] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 02:44:34 (1734680674)
[ 4043.943936] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 4044.939169] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 02:44:35 (1734680675)
[ 4045.808640] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 4046.862423] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 02:44:37 (1734680677)
[ 4047.578440] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 4048.489213] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 02:44:39 (1734680679)
[ 4049.330887] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 4050.089850] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 02:44:41 (1734680681)
[ 4050.810962] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 4051.718718] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 02:44:42 (1734680682)
[ 4052.418221] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 4053.118697] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 02:44:44 (1734680684)
[ 4053.746521] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 4054.435430] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 02:44:45 (1734680685)
[ 4055.073244] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 4055.804300] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 02:44:46 (1734680686)
[ 4056.341779] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 4057.007823] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 02:44:48 (1734680688)
[ 4058.084665] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 4058.806433] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 02:44:50 (1734680690)
[ 4071.226204] Lustre: lustre-MDT0000-mdc-ffff8800a8fb4000: Connection to lustre-MDT0000 (at 192.168.203.104@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4071.231509] Lustre: Skipped 22 previous similar messages
[ 4072.015504] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 02:45:03 (1734680703)
[ 4097.280776] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 02:45:28 (1734680728)
[ 4097.704644] Lustre: *** cfs_fail_loc=51b, val=3***
[ 4105.559373] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 02:45:36 (1734680736)
[ 4105.786533] Lustre: *** cfs_fail_loc=51b, val=4***
[ 4113.325530] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 02:45:44 (1734680744)
[ 4113.511499] Lustre: *** cfs_fail_loc=50f, val=3***
[ 4118.934104] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 02:45:49 (1734680749)
[ 4119.104676] Lustre: *** cfs_fail_loc=50f, val=4***
[ 4123.953012] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 02:45:54 (1734680754)
[ 4124.298516] Lustre: *** cfs_fail_loc=510, val=3***
[ 4129.557732] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 02:46:00 (1734680760)
[ 4129.766051] Lustre: *** cfs_fail_loc=51b, val=3***
[ 4137.082991] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 02:46:08 (1734680768)
[ 4199.337742] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 02:47:10 (1734680830)
[ 4199.537250] LustreError: 25105:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 4202.082548] LustreError: lustre-MDT0000-mdc-ffff8800a8fb4000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4202.111952] LustreError: 25124:0:(file.c:262:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a8fb4000: inode [0x20000a042:0x8:0x0] mdc close failed: rc = -108
[ 4202.121799] LustreError: 25129:0:(file.c:5963:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4202.123745] LustreError: 25124:0:(ldlm_resource.c:1149:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8800a8fb4000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff88009f070e00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4202.148445] LustreError: 25129:0:(file.c:5963:ll_inode_revalidate_fini()) Skipped 20 previous similar messages
[ 4203.556125] LustreError: 25105:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 4205.642552] LustreError: 25142:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 4208.062030] LustreError: lustre-MDT0000-mdc-ffff8800a8fb4000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4208.071160] LustreError: 25161:0:(ldlm_resource.c:1149:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8800a8fb4000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8800a9746000) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4208.085193] LustreError: 25161:0:(ldlm_resource.c:1149:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4209.658796] LustreError: 25142:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 4216.132094] LustreError: lustre-MDT0000-mdc-ffff8800a8fb4000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4216.343453] LustreError: 25221:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 4218.740660] LustreError: lustre-MDT0000-mdc-ffff8800a8fb4000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4218.780912] LustreError: 25245:0:(file.c:5963:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4218.787044] LustreError: 25245:0:(file.c:5963:ll_inode_revalidate_fini()) Skipped 2 previous similar messages
[ 4220.353451] LustreError: 25221:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 4220.367494] LustreError: 25221:0:(file.c:262:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a8fb4000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 4220.376084] LustreError: 25221:0:(file.c:262:ll_close_inode_openhandle()) Skipped 4 previous similar messages
[ 4225.405769] LustreError: 25305:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 4227.945799] LustreError: lustre-MDT0000-mdc-ffff8800a8fb4000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4227.963945] LustreError: 25324:0:(ldlm_resource.c:1149:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8800a8fb4000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff88012e398b00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4227.988501] LustreError: 25324:0:(ldlm_resource.c:1149:ldlm_resource_complain()) Skipped 2 previous similar messages
[ 4229.425137] LustreError: 25305:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 4236.092325] LustreError: lustre-MDT0000-mdc-ffff8800a8fb4000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4236.439550] LustreError: 25384:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 4239.095198] LustreError: lustre-MDT0000-mdc-ffff8800a8fb4000: operation ldlm_enqueue to node 192.168.203.104@tcp failed: rc = -107
[ 4239.101672] LustreError: Skipped 11 previous similar messages
[ 4239.108768] LustreError: lustre-MDT0000-mdc-ffff8800a8fb4000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4239.140289] LustreError: 25404:0:(ldlm_resource.c:1149:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8800a8fb4000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff88009f070500) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4239.146816] LustreError: 25404:0:(ldlm_resource.c:1149:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4240.463171] LustreError: 25384:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 4243.307615] LustreError: lustre-MDT0000-mdc-ffff8800a8fb4000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4244.570670] LustreError: 25418:0:(file.c:262:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a8fb4000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 4252.012595] LustreError: 25497:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 4252.019839] LustreError: 25497:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 4254.746645] LustreError: lustre-MDT0000-mdc-ffff8800a8fb4000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4254.784819] LustreError: 25526:0:(file.c:5963:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4254.789492] LustreError: 25526:0:(file.c:5963:ll_inode_revalidate_fini()) Skipped 55 previous similar messages
[ 4256.048167] LustreError: 25497:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 4256.061171] LustreError: 25497:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 4256.071971] LustreError: 25497:0:(file.c:262:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a8fb4000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 4263.752622] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 02:48:14 (1734680894)
[ 4281.313982] LustreError: 9093:0:(ldlm_lockd.c:2927:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 4281.318991] LustreError: 9093:0:(ldlm_lockd.c:2927:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 4285.343112] LustreError: 9093:0:(ldlm_lockd.c:2927:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 4285.346114] LustreError: 9093:0:(ldlm_lockd.c:2927:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 4289.071657] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 02:48:40 (1734680920)
[ 4334.206894] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 02:49:25 (1734680965)
[ 4390.281267] Lustre: 27906:0:(client.c:2364:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1734680967/real 1734680967]  req@ffff8800a80e4700 x1818940384290304/t0(0) o101->lustre-MDT0000-mdc-ffff8800a8fb4000@192.168.203.104@tcp:12/10 lens 576/1024 e 0 to 1 dl 1734681022 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'stat.0' uid:0 gid:0
[ 4390.304971] Lustre: 27906:0:(client.c:2364:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
[ 4390.331927] Lustre: lustre-MDT0000-mdc-ffff8800a8fb4000: Connection restored to  (at 192.168.203.104@tcp)
[ 4390.334849] Lustre: Skipped 29 previous similar messages
[ 4393.968795] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 02:50:24 (1734681024)
[ 4421.860452] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 02:50:53 (1734681053)
[ 4422.185651] Lustre: Mounted lustre-client
[ 4543.611441] Lustre: Unmounted lustre-client
[ 4546.793255] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 02:52:57 (1734681177)
[ 4547.134534] LustreError: 30399:0:(osc_request.c:2941:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 4000ms
[ 4549.326664] LustreError: lustre-OST0000-osc-ffff8800a8fb4000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4549.332169] LustreError: 30491:0:(ldlm_resource.c:1149:ldlm_resource_complain()) lustre-OST0000-osc-ffff8800a8fb4000: namespace resource [0x240000400:0x7f0d:0x0].0x0 (ffff88009f070600) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4549.339880] LustreError: 30491:0:(ldlm_resource.c:1149:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4551.154119] LustreError: 30399:0:(osc_request.c:2941:osc_build_rpc()) cfs_fail_timeout id 414 awake
[ 4551.164259] Lustre: 1964:0:(llite_lib.c:4127:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.104@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 4554.379584] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 02:53:05 (1734681185)
[ 4614.206109] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 02:54:05 (1734681245)
[ 4615.022878] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 4615.986942] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 02:54:07 (1734681247)
[ 4671.929312] Lustre: lustre-MDT0000-mdc-ffff8800a8fb4000: Connection to lustre-MDT0000 (at 192.168.203.104@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4671.939116] Lustre: Skipped 15 previous similar messages
[ 4675.470615] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 4676.333373] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 02:55:07 (1734681307)
[ 4733.177893] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 02:56:04 (1734681364)
[ 4733.926982] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 4734.737561] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 02:56:05 (1734681365)
[ 4735.802024] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 4736.868790] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 02:56:07 (1734681367)
[ 4752.396216] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 02:56:23 (1734681383)
[ 4757.131172] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4777.364503] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 192.168.203.104@tcp) was lost; in progress operations using this service will fail
[ 4777.382985] LustreError: Skipped 2 previous similar messages
[ 4777.394275] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0xd1ece635b8681370 to 0xd1ece635b8682e27
[ 4777.398411] Lustre: Skipped 2 previous similar messages
[ 4779.362797] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4780.036820] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4785.470867] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 02:56:56 (1734681416)
[ 4786.233354] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 4786.939041] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 02:56:58 (1734681418)
[ 4798.906548] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 02:57:10 (1734681430)
[ 4817.153425] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 02:57:28 (1734681448)
[ 4843.965367] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4845.354241] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4928.653330] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4929.573356] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4952.121303] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4952.963473] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4984.609943] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 03:00:15 (1734681615)
[ 4988.678884] LustreError: lustre-OST0000-osc-ffff8800a8fb4000: operation ost_setattr to node 192.168.203.104@tcp failed: rc = -19
[ 4988.682471] LustreError: Skipped 49 previous similar messages
[ 4990.447100] Lustre: 1965:0:(client.c:2364:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1734681567/real 1734681567]  req@ffff8800a7e7f100 x1818940390848896/t0(0) o400->lustre-MDT0000-mdc-ffff8800a8fb4000@192.168.203.104@tcp:12/10 lens 224/224 e 0 to 1 dl 1734681622 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0
[ 4990.474734] Lustre: 1965:0:(client.c:2364:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
[ 5016.823529] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5018.495207] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5177.233878] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 03:03:28 (1734681808)
[ 5266.501381] Lustre: MGC192.168.203.104@tcp: Connection restored to 192.168.203.104@tcp (at 192.168.203.104@tcp)
[ 5266.510888] Lustre: Skipped 14 previous similar messages
[ 5287.405404] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 03:05:18 (1734681918)
[ 5288.096143] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 5288.964559] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 03:05:20 (1734681920)
[ 5289.875407] Lustre: lustre-MDT0000-mdc-ffff8800a8fb4000: Connection to lustre-MDT0000 (at 192.168.203.104@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5289.881201] Lustre: Skipped 9 previous similar messages
[ 5289.888892] LustreError: lustre-MDT0000-mdc-ffff8800a8fb4000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5293.037955] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 03:05:24 (1734681924)
[ 5458.373838] LustreError: lustre-OST0000-osc-ffff8800a8fb4000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5459.466183] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 03:08:10 (1734682090)
[ 5494.772614] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 03:08:45 (1734682125)
[ 5495.461242] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 5496.408443] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 03:08:47 (1734682127)
[ 5497.067305] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 5498.007956] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 03:08:49 (1734682129)
[ 5499.210911] Lustre: DEBUG MARKER: SKIP: recovery-small test_152 MDS Linux kernel does not support killable semaphore
[ 5500.063031] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 03:08:51 (1734682131)
[ 5531.494240] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 192.168.203.104@tcp) was lost; in progress operations using this service will fail
[ 5531.509377] LustreError: Skipped 5 previous similar messages
[ 5531.547074] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0xd1ece635b877e807 to 0xd1ece635b877f21e
[ 5531.564756] Lustre: Skipped 5 previous similar messages
[ 5537.184743] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 03:09:28 (1734682168)
[ 5537.898472] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 5538.862746] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 03:09:29 (1734682169)
[ 5539.564825] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 5540.539260] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 03:09:31 (1734682171)
[ 5545.202526] Lustre: Unmounted lustre-client
[ 5546.022532] Lustre: Mounted lustre-client
[ 5547.586206] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5568.101098] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: total 2260