[ 1461.578091] LustreError: lustre-OST0000-osc-ffff8800a8f25800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1461.586660] Lustre: 1960:0:(llite_lib.c:4117:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.143@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1464.114537] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 16:23:48 (1746131028)
[ 1464.921739] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1465.524902] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 16:23:49 (1746131029)
[ 1466.188592] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1466.856675] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 16:23:51 (1746131031)
[ 1482.986681] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 192.168.202.143@tcp) was lost; in progress operations using this service will fail
[ 1482.994324] Lustre: Evicted from MGS (at 192.168.202.143@tcp) after server handle changed from 0x694608eac6e26d3a to 0x694608eac6e2950a
[ 1577.616330] LustreError: lustre-MDT0000-mdc-ffff8800a8f25800: operation mds_reint to node 192.168.202.143@tcp failed: rc = -19
[ 1577.624227] LustreError: Skipped 3 previous similar messages
[ 1593.306814] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 192.168.202.143@tcp) was lost; in progress operations using this service will fail
[ 1593.320992] Lustre: Evicted from MGS (at 192.168.202.143@tcp) after server handle changed from 0x694608eac6e2950a to 0x694608eac6e82e89
[ 1600.800198] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 16:26:04 (1746131164)
[ 1601.031927] Lustre: *** cfs_fail_loc=305, val=0***
[ 1601.033801] Lustre: Skipped 4 previous similar messages
[ 1637.015752] LustreError: lustre-MDT0000-mdc-ffff8800a8f25800: operation ldlm_enqueue to node 192.168.202.143@tcp failed: rc = -107
[ 1637.404115] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 192.168.202.143@tcp) was lost; in progress operations using this service will fail
[ 1637.413392] Lustre: Evicted from MGS (at 192.168.202.143@tcp) after server handle changed from 0x694608eac6e82e89 to 0x694608eac6e834be
[ 1638.596638] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1639.235978] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1643.398521] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 16:26:47 (1746131207)
[ 1652.358798] LustreError: lustre-MDT0000-mdc-ffff8800a8f25800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 1652.427004] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 192.168.202.143@tcp) was lost; in progress operations using this service will fail
[ 1652.436161] Lustre: Evicted from MGS (at 192.168.202.143@tcp) after server handle changed from 0x694608eac6e834be to 0x694608eac6e8384c
[ 1660.038431] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 16:27:04 (1746131224)
[ 1666.623355] LustreError: lustre-OST0000-osc-ffff8800a8f25800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1675.711724] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 16:27:19 (1746131239)
[ 1678.968281] hrtimer: interrupt took 3226033 ns
[ 1702.512067] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 192.168.202.143@tcp) was lost; in progress operations using this service will fail
[ 1702.528312] Lustre: Evicted from MGS (at 192.168.202.143@tcp) after server handle changed from 0x694608eac6e8384c to 0x694608eac6e8c0da
[ 1709.568385] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1710.538362] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1773.319164] LustreError: lustre-MDT0000-mdc-ffff8800a8f25800: operation mds_reint to node 192.168.202.143@tcp failed: rc = -19
[ 1773.325863] LustreError: Skipped 2 previous similar messages
[ 1787.709736] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 192.168.202.143@tcp) was lost; in progress operations using this service will fail
[ 1787.721108] Lustre: Evicted from MGS (at 192.168.202.143@tcp) after server handle changed from 0x694608eac6e8c0da to 0x694608eac6ebeabf
[ 1794.437694] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1795.261622] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1872.938186] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 192.168.202.143@tcp) was lost; in progress operations using this service will fail
[ 1872.947074] Lustre: Evicted from MGS (at 192.168.202.143@tcp) after server handle changed from 0x694608eac6ebeabf to 0x694608eac6ef2624
[ 1879.315080] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1879.898182] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1903.344236] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 16:31:07 (1746131467)
[ 1905.599163] LustreError: lustre-MDT0000-mdc-ffff8800a8f25800: operation ldlm_enqueue to node 192.168.202.143@tcp failed: rc = -19
[ 1905.607039] LustreError: Skipped 2 previous similar messages
[ 1922.623966] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 1938.037998] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 192.168.202.143@tcp) was lost; in progress operations using this service will fail
[ 1938.046744] LustreError: Skipped 1 previous similar message
[ 1938.069592] Lustre: Evicted from MGS (at 192.168.202.143@tcp) after server handle changed from 0x694608eac6f15288 to 0x694608eac6f169f0
[ 1938.078323] Lustre: Skipped 1 previous similar message
[ 1941.675078] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 1947.822565] Lustre: lustre-MDT0000-mdc-ffff8800a8f25800: Connection to lustre-MDT0000 (at 192.168.202.143@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 1947.832526] Lustre: Skipped 17 previous similar messages
[ 1963.110672] Lustre: MGC192.168.202.143@tcp: Connection restored to 192.168.202.143@tcp (at 192.168.202.143@tcp)
[ 1963.121924] Lustre: Skipped 26 previous similar messages
[ 1965.298208] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 1994.988969] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2034.509918] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2078.239828] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 192.168.202.143@tcp) was lost; in progress operations using this service will fail
[ 2078.261373] LustreError: Skipped 3 previous similar messages
[ 2078.307823] Lustre: Evicted from MGS (at 192.168.202.143@tcp) after server handle changed from 0x694608eac6f33af2 to 0x694608eac6f449e5
[ 2078.321475] Lustre: Skipped 3 previous similar messages
[ 2079.722754] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2150.298216] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 16:35:14 (1746131714)
[ 2161.779578] LustreError: lustre-OST0000-osc-ffff8800a8f25800: operation ldlm_enqueue to node 192.168.202.143@tcp failed: rc = -107
[ 2161.783176] LustreError: Skipped 12 previous similar messages
[ 2183.665120] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2184.906096] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 2510.056034] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2510.574128] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 2823.391281] LustreError: lustre-OST0000-osc-ffff8800a8f25800: operation ldlm_enqueue to node 192.168.202.143@tcp failed: rc = -107
[ 2823.395671] LustreError: Skipped 1 previous similar message
[ 2823.397575] Lustre: lustre-OST0000-osc-ffff8800a8f25800: Connection to lustre-OST0000 (at 192.168.202.143@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2823.404062] Lustre: Skipped 6 previous similar messages
[ 2842.245187] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2842.705294] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3115.622555] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 16:51:19 (1746132679)
[ 3132.161766] Lustre: 22363:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746132680/real 1746132680]  req@ffff8800a7cd0380 x1830949673306112/t0(0) o101->lustre-MDT0000-mdc-ffff8800a8f25800@192.168.202.143@tcp:12/10 lens 576/1024 e 0 to 1 dl 1746132696 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0
[ 3132.172895] Lustre: 22363:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
[ 3132.191402] Lustre: lustre-MDT0000-mdc-ffff8800a8f25800: Connection restored to 192.168.202.143@tcp (at 192.168.202.143@tcp)
[ 3132.198308] Lustre: Skipped 10 previous similar messages
[ 3134.572605] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 16:51:38 (1746132698)
[ 3154.286311] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 16:51:58 (1746132718)
[ 3173.291277] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 16:52:17 (1746132737)
[ 3173.434107] Lustre: Mounted lustre-client
[ 3198.494903] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 192.168.202.143@tcp) was lost; in progress operations using this service will fail
[ 3198.505639] LustreError: Skipped 1 previous similar message
[ 3198.518266] Lustre: Evicted from MGS (at 192.168.202.143@tcp) after server handle changed from 0x694608eac6f5bcfa to 0x694608eac7468068
[ 3198.525927] Lustre: Skipped 1 previous similar message
[ 3201.387852] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3201.950124] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3203.348013] LustreError: 25359:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 3203.350612] LustreError: 25359:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 3203.368246] Lustre: Unmounted lustre-client
[ 3205.424333] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 16:52:49 (1746132769)
[ 3210.431161] Lustre: 1960:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746132758/real 1746132758]  req@ffff880130610700 x1830949673324544/t0(0) o400->lustre-MDT0000-mdc-ffff8800a8f25800@192.168.202.143@tcp:12/10 lens 224/224 e 0 to 1 dl 1746132774 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0
[ 3210.450429] Lustre: 1960:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 3370.654187] Lustre: 1961:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746132919/real 1746132919]  req@ffff88012f902a00 x1830949673382784/t0(0) o4->lustre-OST0000-osc-ffff8800a8f25800@192.168.202.143@tcp:6/4 lens 488/448 e 0 to 1 dl 1746132935 ref 2 fl Rpc:XQr/202/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0
[ 3370.664546] Lustre: 1961:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 63 previous similar messages
[ 3434.698264] Lustre: lustre-OST0000-osc-ffff8800a8f25800: Connection to lustre-OST0000 (at 192.168.202.143@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3434.710569] Lustre: Skipped 18 previous similar messages
[ 3494.056864] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 16:57:37 (1746133057)
[ 3537.910725] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 16:58:21 (1746133101)
[ 3539.377858] LustreError: 27909:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup
[ 3539.385669] LustreError: 27909:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 3539.408017] Lustre: Unmounted lustre-client
[ 3548.980725] Lustre: Mounted lustre-client
[ 3551.921932] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 16:58:35 (1746133115)
[ 3552.145505] LustreError: 28961:0:(mdc_locks.c:1333:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 3553.234594] Lustre: *** cfs_fail_loc=305, val=0***
[ 3572.181452] LustreError: 28961:0:(mdc_locks.c:1333:mdc_finish_intent_lock()) cfs_fail_timeout id 801 awake
[ 3575.308921] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 16:58:59 (1746133139)
[ 3575.580801] Lustre: Mounted lustre-client
[ 3576.156323] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 3586.188636] LustreError: 29707:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 3586.191846] LustreError: 29707:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages
[ 3586.209442] Lustre: Unmounted lustre-client
[ 3588.738256] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 16:59:12 (1746133152)
[ 3638.518586] LustreError: lustre-MDT0000-mdc-ffff8800b4271800: operation mds_reint to node 192.168.202.143@tcp failed: rc = -19
[ 3653.707849] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 192.168.202.143@tcp) was lost; in progress operations using this service will fail
[ 3653.717968] Lustre: Evicted from MGS (at 192.168.202.143@tcp) after server handle changed from 0x694608eac7468df8 to 0x694608eac7495a67
[ 3734.766303] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 17:01:38 (1746133298)
[ 3737.070958] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 3743.869733] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 192.168.202.143@tcp) was lost; in progress operations using this service will fail
[ 3743.877994] Lustre: Evicted from MGS (at 192.168.202.143@tcp) after server handle changed from 0x694608eac7495a67 to 0x694608eac74dfb3c
[ 3743.882909] Lustre: MGC192.168.202.143@tcp: Connection restored to 192.168.202.143@tcp (at 192.168.202.143@tcp)
[ 3743.887077] Lustre: Skipped 25 previous similar messages
[ 3743.902388] LustreError: lustre-MDT0000-mdc-ffff8800b4271800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 3748.077173] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 17:01:52 (1746133312)
[ 3748.291754] Lustre: Mounted lustre-client
[ 3753.298837] LustreError: lustre-OST0000-osc-ffff8800b4271800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 3764.800201] Lustre: 1963:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746133313/real 1746133313]  req@ffff8800a7a23800 x1830949678023296/t0(0) o103->lustre-OST0000-osc-ffff8800b6fcf000@192.168.202.143@tcp:17/18 lens 328/224 e 0 to 1 dl 1746133329 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0
[ 3764.815400] Lustre: 1963:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 75 previous similar messages
[ 3765.317595] LustreError: 579:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 3765.322204] LustreError: 579:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 3765.357482] Lustre: Unmounted lustre-client
[ 3768.325830] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 17:02:12 (1746133332)
[ 3773.699576] LustreError: lustre-MDT0000-mdc-ffff8800b4271800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 3773.711302] LustreError: 1315:0:(file.c:6135:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 3776.969785] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 17:02:21 (1746133341)
[ 3777.055754] LustreError: 2099:0:(recover.c:322:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 3781.316388] LustreError: lustre-MDT0000-mdc-ffff8800b4271800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 3781.326137] LustreError: 2118:0:(import.c:287:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 3781.330581] LustreError: 2099:0:(recover.c:322:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=729
[ 3781.337848] LustreError: 2099:0:(import.c:709:ptlrpc_connect_import_locked()) already connecting
[ 3781.350887] LustreError: 2123:0:(file.c:6135:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3782.396459] LustreError: 2130:0:(file.c:6135:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3782.400510] LustreError: 2130:0:(file.c:6135:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 3784.487323] LustreError: 2154:0:(file.c:6135:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3784.491597] LustreError: 2154:0:(file.c:6135:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 3788.704636] LustreError: 2201:0:(file.c:6135:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3788.709521] LustreError: 2201:0:(file.c:6135:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 3794.246516] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 17:02:38 (1746133358)
[ 3815.149426] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3815.817517] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3820.773358] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 17:03:04 (1746133384)
[ 3839.649131] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3840.312012] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3845.144485] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 17:03:29 (1746133409)
[ 3891.173644] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3891.885571] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3895.524739] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 17:04:19 (1746133459)
[ 3915.489676] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3916.093566] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3918.546291] LustreError: 8239:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup
[ 3918.548619] LustreError: 8239:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 3918.565651] Lustre: Unmounted lustre-client
[ 3944.833768] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3946.664741] Lustre: Mounted lustre-client
[ 3949.114734] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 17:05:13 (1746133513)
[ 3950.310399] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 3951.402333] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 17:05:15 (1746133515)
[ 3966.075195] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 17:05:30 (1746133530)
[ 3966.939411] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 3967.651869] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 17:05:31 (1746133531)
[ 3968.357637] Lustre: *** cfs_fail_loc=805, val=0***
[ 3968.382302] Lustre: Mounted lustre-client
[ 3971.028477] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 3988.417013] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 192.168.202.143@tcp) was lost; in progress operations using this service will fail
[ 3988.445271] Lustre: Evicted from MGS (at 192.168.202.143@tcp) after server handle changed from 0x694608eac74e0522 to 0x694608eac74e08a2
[ 3989.423559] LustreError: 11764:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 3989.426432] LustreError: 11764:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages
[ 3989.452154] Lustre: Unmounted lustre-client
[ 3993.159064] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 17:05:57 (1746133557)
[ 4010.208095] LustreError: 1959:0:(client.c:3262:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff8800a8979c00 x1830949678090880/t94489280516(94489280516) o101->lustre-MDT0000-mdc-ffff8800b5977800@192.168.202.143@tcp:12/10 lens 664/608 e 0 to 0 dl 1746133590 ref 2 fl Interpret:RPQU/204/0 rc 301/301 job:'multiop.0' uid:0 gid:0
[ 4013.598154] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4014.157833] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4015.514935] checkstat (13267) used greatest stack depth: 10624 bytes left
[ 4017.725669] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 17:06:21 (1746133581)
[ 4018.324573] LustreError: lustre-OST0000-osc-ffff8800b5977800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4018.332363] Lustre: 1962:0:(llite_lib.c:4117:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.143@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 4018.343611] LustreError: 13927:0:(ldlm_resource.c:982:ldlm_resource_complain()) lustre-OST0000-osc-ffff8800b5977800: namespace resource [0x240000400:0xb502:0x0].0x0 (ffff8800b6694100) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4022.226863] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 17:06:26 (1746133586)
[ 4022.823413] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 4023.467311] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 17:06:27 (1746133587)
[ 4024.008923] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 4024.643748] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 17:06:28 (1746133588)
[ 4025.356955] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 4026.047535] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 17:06:30 (1746133590)
[ 4026.663474] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 4027.429818] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 17:06:31 (1746133591)
[ 4028.142140] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 4028.920267] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 17:06:33 (1746133593)
[ 4029.518525] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 4030.337134] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 17:06:34 (1746133594)
[ 4030.986735] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 4031.754752] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 17:06:35 (1746133595)
[ 4032.437474] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 4033.177309] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 17:06:37 (1746133597)
[ 4033.839556] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 4034.634276] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 17:06:38 (1746133598)
[ 4035.372753] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 4036.243725] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 17:06:40 (1746133600)
[ 4036.956205] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 4037.731588] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 17:06:41 (1746133601)
[ 4038.926314] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 4039.808978] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 17:06:43 (1746133603)
[ 4048.506981] Lustre: lustre-MDT0000-mdc-ffff8800b5977800: Connection to lustre-MDT0000 (at 192.168.202.143@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4048.512792] Lustre: Skipped 22 previous similar messages
[ 4053.025977] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 17:06:57 (1746133617)
[ 4076.092309] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 17:07:20 (1746133640)
[ 4076.305573] Lustre: *** cfs_fail_loc=51b, val=3***
[ 4083.229645] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 17:07:27 (1746133647)
[ 4083.526472] Lustre: *** cfs_fail_loc=51b, val=4***
[ 4090.121773] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 17:07:34 (1746133654)
[ 4090.219720] Lustre: *** cfs_fail_loc=50f, val=3***
[ 4093.773656] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 17:07:38 (1746133658)
[ 4093.873225] Lustre: *** cfs_fail_loc=50f, val=4***
[ 4097.581819] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 17:07:41 (1746133661)
[ 4097.717809] Lustre: *** cfs_fail_loc=510, val=3***
[ 4101.483713] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 17:07:45 (1746133665)
[ 4101.635397] Lustre: *** cfs_fail_loc=51b, val=3***
[ 4108.757678] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 17:07:52 (1746133672)
[ 4169.973406] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 17:08:54 (1746133734)
[ 4170.121518] LustreError: 25093:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 4172.422320] LustreError: lustre-MDT0000-mdc-ffff8800b5977800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4172.433908] LustreError: 25109:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800b5977800: inode [0x20000a042:0x8:0x0] mdc close failed: rc = -108
[ 4172.448529] LustreError: 25109:0:(ldlm_resource.c:982:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8800b5977800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff88012b35c300) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4172.452811] LustreError: 25114:0:(file.c:6135:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4172.452814] LustreError: 25114:0:(file.c:6135:ll_inode_revalidate_fini()) Skipped 20 previous similar messages
[ 4174.131297] LustreError: 25093:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 4176.158614] LustreError: 25138:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 4178.485874] LustreError: lustre-MDT0000-mdc-ffff8800b5977800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4178.494581] LustreError: 25154:0:(ldlm_resource.c:982:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8800b5977800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8800a7cf4600) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4178.502841] LustreError: 25154:0:(ldlm_resource.c:982:ldlm_resource_complain()) Skipped 2 previous similar messages
[ 4180.166572] LustreError: 25138:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 4186.616822] LustreError: lustre-MDT0000-mdc-ffff8800b5977800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4186.810784] LustreError: 25201:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 4189.206521] LustreError: lustre-MDT0000-mdc-ffff8800b5977800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4189.240030] LustreError: 25222:0:(file.c:6135:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4189.246857] LustreError: 25222:0:(file.c:6135:ll_inode_revalidate_fini()) Skipped 2 previous similar messages
[ 4190.819211] LustreError: 25201:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 4190.828216] LustreError: 25201:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800b5977800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 4190.835416] LustreError: 25201:0:(file.c:247:ll_close_inode_openhandle()) Skipped 4 previous similar messages
[ 4195.819030] LustreError: 25281:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 4198.298503] LustreError: lustre-MDT0000-mdc-ffff8800b5977800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4198.320282] LustreError: 25297:0:(ldlm_resource.c:982:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8800b5977800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff88012b35cf00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4198.346495] LustreError: 25297:0:(ldlm_resource.c:982:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4199.842905] LustreError: 25281:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 4206.342357] LustreError: lustre-MDT0000-mdc-ffff8800b5977800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4206.537541] LustreError: 25343:0:(ldlm_flock.c:856:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 4208.994639] LustreError: lustre-MDT0000-mdc-ffff8800b5977800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4209.005897] LustreError: 25359:0:(ldlm_resource.c:982:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8800b5977800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8800a7cf4f00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4209.016233] LustreError: 25359:0:(ldlm_resource.c:982:ldlm_resource_complain()) Skipped 2 previous similar messages
[ 4210.543170] LustreError: 25343:0:(ldlm_flock.c:856:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 4212.987638] LustreError: lustre-MDT0000-mdc-ffff8800b5977800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4214.615680] LustreError: 25374:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800b5977800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 4221.533297] LustreError: 25449:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 4221.538357] LustreError: 25449:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 4223.937626] LustreError: lustre-MDT0000-mdc-ffff8800b5977800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4223.979145] LustreError: 25475:0:(file.c:6135:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4223.983876] LustreError: 25475:0:(file.c:6135:ll_inode_revalidate_fini()) Skipped 55 previous similar messages
[ 4225.560202] LustreError: 25449:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800b5977800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 4227.958154] LustreError: 25470:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 4227.964407] LustreError: 25470:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) Skipped 2 previous similar messages
[ 4231.450412] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 17:09:55 (1746133795)
[ 4248.955625] LustreError: 20100:0:(ldlm_lockd.c:2901:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 4248.975502] LustreError: 20100:0:(ldlm_lockd.c:2901:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 4257.277253] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 17:10:21 (1746133821)
[ 4303.566040] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 17:11:07 (1746133867)
[ 4359.790394] Lustre: lustre-MDT0000-mdc-ffff8800b5977800: Connection restored to 192.168.202.143@tcp (at 192.168.202.143@tcp)
[ 4359.795090] Lustre: Skipped 30 previous similar messages
[ 4362.833852] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 17:12:06 (1746133926)
[ 4390.120848] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 17:12:34 (1746133954)
[ 4390.542403] Lustre: Mounted lustre-client
[ 4511.930834] LustreError: 29629:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 4511.934644] LustreError: 29629:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 4511.957634] Lustre: Unmounted lustre-client
[ 4514.654195] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 17:14:38 (1746134078)
[ 4514.993517] LustreError: 30305:0:(osc_request.c:2943:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 4000ms
[ 4517.037627] LustreError: lustre-OST0000-osc-ffff8800b5977800: operation ost_statfs to node 192.168.202.143@tcp failed: rc = -107
[ 4517.041508] LustreError: Skipped 12 previous similar messages
[ 4517.047385] LustreError: lustre-OST0000-osc-ffff8800b5977800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4517.053353] LustreError: 30394:0:(ldlm_resource.c:982:ldlm_resource_complain()) lustre-OST0000-osc-ffff8800b5977800: namespace resource [0x240000400:0xb52d:0x0].0x0 (ffff8800b6ccbc00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4517.061438] LustreError: 30394:0:(ldlm_resource.c:982:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4519.001153] LustreError: 30305:0:(osc_request.c:2943:osc_build_rpc()) cfs_fail_timeout id 414 awake
[ 4519.003639] LustreError: 30305:0:(osc_request.c:2943:osc_build_rpc()) Skipped 1 previous similar message
[ 4519.006901] Lustre: 1960:0:(llite_lib.c:4117:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.143@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 4521.835138] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 17:14:45 (1746134085)
[ 4578.549595] Lustre: 31033:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746134088/real 1746134088]  req@ffff88009ed72d80 x1830949678257152/t0(0) o101->lustre-MDT0000-mdc-ffff8800b5977800@192.168.202.143@tcp:12/10 lens 328/344 e 0 to 1 dl 1746134143 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0
[ 4578.590268] Lustre: 31033:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
[ 4582.008256] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 17:15:45 (1746134145)
[ 4582.924842] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 4583.800615] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 17:15:47 (1746134147)
[ 4643.126133] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 4644.169308] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 17:16:48 (1746134208)
[ 4700.092358] Lustre: lustre-MDT0000-mdc-ffff8800b5977800: Connection to lustre-MDT0000 (at 192.168.202.143@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4700.101893] Lustre: Skipped 16 previous similar messages
[ 4700.954470] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 17:17:44 (1746134264)
[ 4701.927814] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 4702.994956] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 17:17:46 (1746134266)
[ 4703.858048] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 4705.063934] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 17:17:48 (1746134268)
[ 4723.836556] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 17:18:07 (1746134287)
[ 4730.350022] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4750.194339] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 192.168.202.143@tcp) was lost; in progress operations using this service will fail
[ 4750.209146] LustreError: Skipped 2 previous similar messages
[ 4750.218740] Lustre: Evicted from MGS (at 192.168.202.143@tcp) after server handle changed from 0x694608eac74e12ff to 0x694608eac74e2cf9
[ 4750.225814] Lustre: Skipped 2 previous similar messages
[ 4755.524996] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4756.472965] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4762.702419] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 17:18:46 (1746134326)
[ 4763.904561] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 4764.793946] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 17:18:48 (1746134328)
[ 4778.177767] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 17:19:02 (1746134342)
[ 4800.996953] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 17:19:24 (1746134364)
[ 4830.716127] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4832.122838] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4915.639820] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4916.405335] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4940.050233] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4941.150782] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4968.919896] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 17:22:12 (1746134532)
[ 4999.251259] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5001.673986] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5156.902369] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 17:25:20 (1746134720)
[ 5250.247107] Lustre: MGC192.168.202.143@tcp: Connection restored to 192.168.202.143@tcp (at 192.168.202.143@tcp)
[ 5250.250464] Lustre: Skipped 14 previous similar messages
[ 5272.912389] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 17:27:16 (1746134836)
[ 5273.776729] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 5274.671681] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 17:27:18 (1746134838)
[ 5275.545365] LustreError: lustre-MDT0000-mdc-ffff8800b5977800: operation ldlm_enqueue to node 192.168.202.143@tcp failed: rc = -107
[ 5275.552176] LustreError: Skipped 256 previous similar messages
[ 5275.560552] LustreError: lustre-MDT0000-mdc-ffff8800b5977800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5278.976733] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 17:27:23 (1746134843)
[ 5445.226552] Lustre: Evicted from lustre-OST0000_UUID (at 192.168.202.143@tcp) after server handle changed from 0x694608eac75deafa to 0x694608eac75dec04
[ 5445.231669] Lustre: Skipped 5 previous similar messages
[ 5445.233757] LustreError: lustre-OST0000-osc-ffff8800b5977800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5446.411264] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 17:30:10 (1746135010)
[ 5469.284336] Lustre: 1961:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746135013/real 1746135013]  req@ffff88009eb52d80 x1830949695964544/t0(0) o4->lustre-OST0000-osc-ffff8800b5977800@192.168.202.143@tcp:6/4 lens 4584/448 e 0 to 1 dl 1746135033 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0
[ 5469.298661] Lustre: 1961:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 15 previous similar messages
[ 5469.305913] Lustre: lustre-OST0000-osc-ffff8800b5977800: Connection to lustre-OST0000 (at 192.168.202.143@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5469.323111] Lustre: Skipped 10 previous similar messages
[ 5484.368322] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 17:30:48 (1746135048)
[ 5485.557534] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 5486.870493] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 17:30:50 (1746135050)
[ 5488.030803] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 5489.312291] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 17:30:52 (1746135052)
[ 5491.182761] Lustre: DEBUG MARKER: SKIP: recovery-small test_152 MDS Linux kernel does not support killable semaphore
[ 5492.414164] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 17:30:56 (1746135056)
[ 5525.424540] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 192.168.202.143@tcp) was lost; in progress operations using this service will fail
[ 5525.429432] LustreError: Skipped 5 previous similar messages
[ 5531.490997] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 17:31:35 (1746135095)
[ 5532.332302] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 5533.236563] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 17:31:37 (1746135097)
[ 5534.137381] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 5535.155454] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 17:31:39 (1746135099)
[ 5539.632014] LustreError: 18339:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup
[ 5539.634885] LustreError: 18339:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 5539.703602] Lustre: Unmounted lustre-client
[ 5540.522008] Lustre: Mounted lustre-client
[ 5542.247145] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5562.065754] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: total 2249