[ 1471.581780] LustreError: lustre-OST0000-osc-ffff8800a861e000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1471.592135] Lustre: 1967:0:(llite_lib.c:4118:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.144@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1474.932556] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 21:20:20 (1746148820)
[ 1475.782049] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1476.731306] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 21:20:22 (1746148822)
[ 1477.320675] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1478.010696] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 21:20:24 (1746148824)
[ 1496.666983] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 192.168.203.144@tcp) was lost; in progress operations using this service will fail
[ 1496.683020] Lustre: Evicted from MGS (at 192.168.203.144@tcp) after server handle changed from 0xed7431b5be0f44f3 to 0xed7431b5be0f6a46
[ 1587.570704] LustreError: lustre-MDT0000-mdc-ffff8800a861e000: operation mds_reint to node 192.168.203.144@tcp failed: rc = -19
[ 1587.575320] LustreError: Skipped 2 previous similar messages
[ 1601.942501] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 192.168.203.144@tcp) was lost; in progress operations using this service will fail
[ 1601.960343] Lustre: Evicted from MGS (at 192.168.203.144@tcp) after server handle changed from 0xed7431b5be0f6a46 to 0xed7431b5be14a79f
[ 1609.901411] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 21:22:35 (1746148955)
[ 1610.121931] Lustre: *** cfs_fail_loc=305, val=0***
[ 1610.123897] Lustre: Skipped 4 previous similar messages
[ 1648.088844] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 192.168.203.144@tcp) was lost; in progress operations using this service will fail
[ 1648.103390] Lustre: Evicted from MGS (at 192.168.203.144@tcp) after server handle changed from 0xed7431b5be14a79f to 0xed7431b5be14aca0
[ 1651.477621] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1652.042988] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1655.920401] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 21:23:22 (1746149002)
[ 1663.106046] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 192.168.203.144@tcp) was lost; in progress operations using this service will fail
[ 1663.116566] Lustre: Evicted from MGS (at 192.168.203.144@tcp) after server handle changed from 0xed7431b5be14aca0 to 0xed7431b5be14b17e
[ 1663.132662] LustreError: lustre-MDT0000-mdc-ffff8800a861e000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 1670.544887] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 21:23:36 (1746149016)
[ 1686.313997] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 21:23:52 (1746149032)
[ 1697.369129] LustreError: lustre-MDT0000-mdc-ffff8800a861e000: operation ldlm_enqueue to node 192.168.203.144@tcp failed: rc = -19
[ 1697.376459] LustreError: Skipped 1 previous similar message
[ 1713.182340] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 192.168.203.144@tcp) was lost; in progress operations using this service will fail
[ 1713.213550] Lustre: Evicted from MGS (at 192.168.203.144@tcp) after server handle changed from 0xed7431b5be14b17e to 0xed7431b5be155572
[ 1717.439412] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1718.553203] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1798.397139] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 192.168.203.144@tcp) was lost; in progress operations using this service will fail
[ 1798.406306] Lustre: Evicted from MGS (at 192.168.203.144@tcp) after server handle changed from 0xed7431b5be155572 to 0xed7431b5be188a16
[ 1801.844927] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1802.938978] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1865.685759] LustreError: lustre-MDT0000-mdc-ffff8800a861e000: operation mds_close to node 192.168.203.144@tcp failed: rc = -19
[ 1865.691934] LustreError: Skipped 4 previous similar messages
[ 1883.538561] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 192.168.203.144@tcp) was lost; in progress operations using this service will fail
[ 1883.572934] Lustre: Evicted from MGS (at 192.168.203.144@tcp) after server handle changed from 0xed7431b5be188a16 to 0xed7431b5be1bb48e
[ 1885.926285] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1886.780851] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1911.314628] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 21:27:37 (1746149257)
[ 1930.570641] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 1948.638481] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 192.168.203.144@tcp) was lost; in progress operations using this service will fail
[ 1948.652193] LustreError: Skipped 1 previous similar message
[ 1948.674247] Lustre: Evicted from MGS (at 192.168.203.144@tcp) after server handle changed from 0xed7431b5be1d100a to 0xed7431b5be1d1233
[ 1948.682648] Lustre: Skipped 1 previous similar message
[ 1948.686597] Lustre: MGC192.168.203.144@tcp: Connection restored to 192.168.203.144@tcp (at 192.168.203.144@tcp)
[ 1948.696973] Lustre: Skipped 22 previous similar messages
[ 1949.534572] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 1971.920078] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 1999.893591] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2020.646343] Lustre: lustre-MDT0000-mdc-ffff8800a861e000: Connection to lustre-MDT0000 (at 192.168.203.144@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2020.662286] Lustre: Skipped 16 previous similar messages
[ 2038.436471] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2078.835167] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 192.168.203.144@tcp) was lost; in progress operations using this service will fail
[ 2078.842349] LustreError: Skipped 3 previous similar messages
[ 2078.849301] Lustre: Evicted from MGS (at 192.168.203.144@tcp) after server handle changed from 0xed7431b5be1e0d92 to 0xed7431b5be1f8c85
[ 2078.854873] Lustre: Skipped 3 previous similar messages
[ 2081.856982] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2151.748603] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 21:31:37 (1746149497)
[ 2162.979764] LustreError: lustre-OST0000-osc-ffff8800a861e000: operation ldlm_enqueue to node 192.168.203.144@tcp failed: rc = -107
[ 2162.988200] LustreError: Skipped 5 previous similar messages
[ 2182.156950] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2182.907445] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 2515.483805] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2516.106325] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 2824.849761] LustreError: lustre-OST0000-osc-ffff8800a861e000: operation ldlm_enqueue to node 192.168.203.144@tcp failed: rc = -19
[ 2824.858774] LustreError: Skipped 1 previous similar message
[ 2824.866576] Lustre: lustre-OST0000-osc-ffff8800a861e000: Connection to lustre-OST0000 (at 192.168.203.144@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2824.886279] Lustre: Skipped 4 previous similar messages
[ 2846.040188] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2846.929146] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3117.778472] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 21:47:43 (1746150463)
[ 3134.555328] Lustre: 22353:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746150465/real 1746150465]  req@ffff8800b6723b80 x1830968297631488/t0(0) o101->lustre-MDT0000-mdc-ffff8800a861e000@192.168.203.144@tcp:12/10 lens 576/1024 e 0 to 1 dl 1746150481 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0
[ 3134.569732] Lustre: 22353:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
[ 3134.595245] Lustre: lustre-MDT0000-mdc-ffff8800a861e000: Connection restored to 192.168.203.144@tcp (at 192.168.203.144@tcp)
[ 3134.599249] Lustre: Skipped 9 previous similar messages
[ 3137.886207] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 21:48:03 (1746150483)
[ 3158.299577] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 21:48:24 (1746150504)
[ 3177.840106] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 21:48:44 (1746150524)
[ 3178.017801] Lustre: Mounted lustre-client
[ 3203.062326] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 192.168.203.144@tcp) was lost; in progress operations using this service will fail
[ 3203.071538] LustreError: Skipped 1 previous similar message
[ 3203.082939] Lustre: Evicted from MGS (at 192.168.203.144@tcp) after server handle changed from 0xed7431b5be210184 to 0xed7431b5be54291f
[ 3203.089236] Lustre: Skipped 1 previous similar message
[ 3208.196090] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3208.872753] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3210.364475] LustreError: 25357:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 3210.366618] LustreError: 25357:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 3210.385063] Lustre: Unmounted lustre-client
[ 3212.789744] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 21:49:19 (1746150559)
[ 3219.079286] Lustre: 1968:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746150544/real 1746150544]  req@ffff8800a7c76d80 x1830968297654528/t0(0) o400->lustre-MDT0000-mdc-ffff8800a861e000@192.168.203.144@tcp:12/10 lens 224/224 e 0 to 1 dl 1746150560 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0
[ 3219.102674] Lustre: 1968:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
[ 3379.674523] Lustre: 1966:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746150710/real 1746150710]  req@ffff8800a7c76300 x1830968297700992/t0(0) o4->lustre-OST0000-osc-ffff8800a861e000@192.168.203.144@tcp:6/4 lens 488/448 e 0 to 1 dl 1746150726 ref 2 fl Rpc:XQr/202/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0
[ 3379.689474] Lustre: 1966:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 64 previous similar messages
[ 3427.715478] Lustre: lustre-OST0000-osc-ffff8800a861e000: Connection to lustre-OST0000 (at 192.168.203.144@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3427.734950] Lustre: Skipped 17 previous similar messages
[ 3504.840596] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 21:54:10 (1746150850)
[ 3548.986323] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 21:54:55 (1746150895)
[ 3550.586199] LustreError: 27910:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup
[ 3550.590125] LustreError: 27910:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 3550.623375] Lustre: Unmounted lustre-client
[ 3563.230117] Lustre: Mounted lustre-client
[ 3566.967384] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 21:55:12 (1746150912)
[ 3567.327489] LustreError: 28966:0:(mdc_locks.c:1333:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 3568.336489] LustreError: 28966:0:(mdc_locks.c:1333:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 3568.454640] Lustre: *** cfs_fail_loc=305, val=0***
[ 3587.976462] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 21:55:33 (1746150933)
[ 3588.274378] Lustre: Mounted lustre-client
[ 3589.018459] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 3599.061726] LustreError: 29709:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 3599.075753] LustreError: 29709:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages
[ 3599.112932] Lustre: Unmounted lustre-client
[ 3604.024793] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 21:55:49 (1746150949)
[ 3655.919505] LustreError: lustre-MDT0000-mdc-ffff8800a9820800: operation mds_reint to node 192.168.203.144@tcp failed: rc = -19
[ 3674.398751] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 192.168.203.144@tcp) was lost; in progress operations using this service will fail
[ 3674.407547] Lustre: Evicted from MGS (at 192.168.203.144@tcp) after server handle changed from 0xed7431b5be5439fe to 0xed7431b5be570405
[ 3752.089201] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 21:58:18 (1746151098)
[ 3754.855080] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 3764.556019] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 192.168.203.144@tcp) was lost; in progress operations using this service will fail
[ 3764.569253] Lustre: Evicted from MGS (at 192.168.203.144@tcp) after server handle changed from 0xed7431b5be570405 to 0xed7431b5be5ba948
[ 3764.588193] Lustre: MGC192.168.203.144@tcp: Connection restored to  (at 192.168.203.144@tcp)
[ 3764.592577] LustreError: lustre-MDT0000-mdc-ffff8800a9820800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 3764.605181] Lustre: Skipped 25 previous similar messages
[ 3770.272215] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 21:58:36 (1746151116)
[ 3770.533694] Lustre: Mounted lustre-client
[ 3777.266109] LustreError: lustre-OST0000-osc-ffff8800a9820800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 3786.548531] Lustre: 1968:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746151117/real 1746151117]  req@ffff8800a7aa4e00 x1830968302355328/t0(0) o103->lustre-OST0000-osc-ffff880136820800@192.168.203.144@tcp:17/18 lens 328/224 e 0 to 1 dl 1746151133 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0
[ 3786.583695] Lustre: 1968:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 76 previous similar messages
[ 3787.123806] LustreError: 590:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 3787.127335] LustreError: 590:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 3787.158623] Lustre: Unmounted lustre-client
[ 3790.393271] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 21:58:56 (1746151136)
[ 3795.612763] LustreError: lustre-MDT0000-mdc-ffff8800a9820800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 3795.620615] LustreError: 1326:0:(file.c:6136:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 3799.585094] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 21:59:05 (1746151145)
[ 3799.706733] LustreError: 2110:0:(recover.c:322:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 3803.094967] LustreError: lustre-MDT0000-mdc-ffff8800a9820800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 3803.110317] LustreError: 2128:0:(import.c:287:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 3803.123640] LustreError: 2110:0:(recover.c:322:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=1587
[ 3803.130179] LustreError: 2110:0:(import.c:709:ptlrpc_connect_import_locked()) already connecting
[ 3803.156135] LustreError: 2133:0:(file.c:6136:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3804.208884] LustreError: 2140:0:(file.c:6136:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3804.214258] LustreError: 2140:0:(file.c:6136:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 3806.346979] LustreError: 2164:0:(file.c:6136:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3806.353665] LustreError: 2164:0:(file.c:6136:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 3810.616754] LustreError: 2211:0:(file.c:6136:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3810.622536] LustreError: 2211:0:(file.c:6136:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 3816.362120] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 21:59:22 (1746151162)
[ 3838.003684] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3838.651536] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3843.677462] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 21:59:49 (1746151189)
[ 3866.365855] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3867.180406] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3873.778505] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 22:00:19 (1746151219)
[ 3922.260852] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3923.075486] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3927.849150] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 22:01:13 (1746151273)
[ 3952.081435] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3953.028940] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3956.829350] LustreError: 8263:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup
[ 3956.839447] LustreError: 8263:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 3956.894759] Lustre: Unmounted lustre-client
[ 3986.265041] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3987.928444] Lustre: Mounted lustre-client
[ 3990.712255] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 22:02:16 (1746151336)
[ 3991.869953] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 3992.665500] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 22:02:18 (1746151338)
[ 4004.842321] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 22:02:31 (1746151351)
[ 4005.454968] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 4006.133254] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 22:02:32 (1746151352)
[ 4006.879708] Lustre: *** cfs_fail_loc=805, val=0***
[ 4006.903475] Lustre: Mounted lustre-client
[ 4009.470965] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4026.934677] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 192.168.203.144@tcp) was lost; in progress operations using this service will fail
[ 4026.944458] LustreError: 1964:0:(ldlm_resource.c:982:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8800a8694000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff8800a8e26e00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4026.966196] Lustre: Evicted from MGS (at 192.168.203.144@tcp) after server handle changed from 0xed7431b5be5bb34a to 0xed7431b5be5bb6d1
[ 4027.608586] LustreError: 11790:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 4027.615558] LustreError: 11790:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages
[ 4027.636776] Lustre: Unmounted lustre-client
[ 4031.189078] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 22:02:57 (1746151377)
[ 4047.750997] Lustre: lustre-MDT0000-mdc-ffff8800a8696800: Connection to lustre-MDT0000 (at 192.168.203.144@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4047.760464] Lustre: Skipped 21 previous similar messages
[ 4047.787622] LustreError: 1964:0:(client.c:3262:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff88012b8be680 x1830968302423424/t81604378629(81604378629) o101->lustre-MDT0000-mdc-ffff8800a8696800@192.168.203.144@tcp:12/10 lens 648/608 e 0 to 0 dl 1746151410 ref 2 fl Interpret:RPQU/204/0 rc 301/301 job:'multiop.0' uid:0 gid:0
[ 4053.005329] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4053.744994] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4058.161065] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 22:03:24 (1746151404)
[ 4058.825331] LustreError: lustre-OST0000-osc-ffff8800a8696800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4058.837223] Lustre: 1967:0:(llite_lib.c:4118:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.144@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 4062.814275] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 22:03:28 (1746151408)
[ 4063.393705] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 4064.028423] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 22:03:30 (1746151410)
[ 4064.620786] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 4065.237844] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 22:03:31 (1746151411)
[ 4065.880764] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 4066.551467] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 22:03:32 (1746151412)
[ 4067.027859] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 4067.585388] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 22:03:33 (1746151413)
[ 4068.145812] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 4068.794700] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 22:03:35 (1746151415)
[ 4069.352508] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 4070.014392] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 22:03:36 (1746151416)
[ 4070.621683] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 4071.252123] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 22:03:37 (1746151417)
[ 4071.805757] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 4072.434792] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 22:03:38 (1746151418)
[ 4073.035278] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 4073.742651] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 22:03:39 (1746151419)
[ 4074.348724] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 4075.143751] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 22:03:41 (1746151421)
[ 4075.956710] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 4076.763035] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 22:03:42 (1746151422)
[ 4078.098031] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 4079.081495] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 22:03:45 (1746151425)
[ 4092.344986] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 22:03:58 (1746151438)
[ 4116.845381] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 22:04:22 (1746151462)
[ 4117.094766] Lustre: *** cfs_fail_loc=51b, val=3***
[ 4123.808563] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 22:04:29 (1746151469)
[ 4123.972674] Lustre: *** cfs_fail_loc=51b, val=4***
[ 4130.585542] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 22:04:36 (1746151476)
[ 4130.789636] Lustre: *** cfs_fail_loc=50f, val=3***
[ 4135.546759] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 22:04:41 (1746151481)
[ 4135.729950] Lustre: *** cfs_fail_loc=50f, val=4***
[ 4140.259619] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 22:04:46 (1746151486)
[ 4140.458775] Lustre: *** cfs_fail_loc=510, val=3***
[ 4145.273589] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 22:04:51 (1746151491)
[ 4145.593308] Lustre: *** cfs_fail_loc=51b, val=3***
[ 4152.602768] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 22:04:58 (1746151498)
[ 4214.526637] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 22:06:00 (1746151560)
[ 4214.765334] LustreError: 25126:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 4217.121732] LustreError: lustre-MDT0000-mdc-ffff8800a8696800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4217.130115] LustreError: 25142:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a8696800: inode [0x20000a042:0x8:0x0] mdc close failed: rc = -108
[ 4217.145141] LustreError: 25142:0:(ldlm_resource.c:982:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8800a8696800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff88012a5a0400) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4217.155817] LustreError: 25147:0:(file.c:6136:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4217.155819] LustreError: 25147:0:(file.c:6136:ll_inode_revalidate_fini()) Skipped 20 previous similar messages
[ 4217.173336] LustreError: 25142:0:(ldlm_resource.c:982:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4218.785243] LustreError: 25126:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 4220.830333] LustreError: 25171:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 4223.286486] LustreError: lustre-MDT0000-mdc-ffff8800a8696800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4223.296096] LustreError: 25187:0:(ldlm_resource.c:982:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8800a8696800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8800b5d38200) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4223.308961] LustreError: 25187:0:(ldlm_resource.c:982:ldlm_resource_complain()) Skipped 2 previous similar messages
[ 4224.844278] LustreError: 25171:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 4231.301986] LustreError: lustre-MDT0000-mdc-ffff8800a8696800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4231.307891] LustreError: 25220:0:(ldlm_resource.c:982:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8800a8696800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff88012a5a0900) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4231.316206] LustreError: 25220:0:(ldlm_resource.c:982:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4231.501699] LustreError: 25233:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 4233.892771] LustreError: lustre-MDT0000-mdc-ffff8800a8696800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4233.922380] LustreError: 25254:0:(file.c:6136:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4233.927178] LustreError: 25254:0:(file.c:6136:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4235.510249] LustreError: 25233:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 4235.516799] LustreError: 25233:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a8696800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 4235.524482] LustreError: 25233:0:(file.c:247:ll_close_inode_openhandle()) Skipped 4 previous similar messages
[ 4240.352841] LustreError: 25314:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 4242.731478] LustreError: lustre-MDT0000-mdc-ffff8800a8696800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4242.740810] LustreError: 25331:0:(ldlm_resource.c:982:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8800a8696800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8800a1f91900) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4244.372264] LustreError: 25314:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 4250.885076] LustreError: lustre-MDT0000-mdc-ffff8800a8696800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4251.117106] LustreError: 25377:0:(ldlm_flock.c:856:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 4253.579681] LustreError: lustre-MDT0000-mdc-ffff8800a8696800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4253.596749] LustreError: 25393:0:(ldlm_resource.c:982:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8800a8696800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff88012a5a0800) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4253.609554] LustreError: 25393:0:(ldlm_resource.c:982:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4255.123452] LustreError: 25377:0:(ldlm_flock.c:856:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 4257.694223] LustreError: lustre-MDT0000-mdc-ffff8800a8696800: operation ldlm_enqueue to node 192.168.203.144@tcp failed: rc = -107
[ 4257.699621] LustreError: Skipped 12 previous similar messages
[ 4257.710024] LustreError: lustre-MDT0000-mdc-ffff8800a8696800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4259.209696] LustreError: 25408:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a8696800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 4266.182897] LustreError: 25483:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 4266.187111] LustreError: 25483:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 4268.616577] LustreError: lustre-MDT0000-mdc-ffff8800a8696800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4268.640456] LustreError: 25509:0:(file.c:6136:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4268.644920] LustreError: 25509:0:(file.c:6136:ll_inode_revalidate_fini()) Skipped 53 previous similar messages
[ 4270.194980] LustreError: 25483:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 4270.199917] LustreError: 25483:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 4270.204857] LustreError: 25483:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a8696800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 4276.041334] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 22:07:02 (1746151622)
[ 4293.391026] LustreError: 9123:0:(ldlm_lockd.c:2901:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 4293.394820] LustreError: 9123:0:(ldlm_lockd.c:2901:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 4297.408256] LustreError: 9123:0:(ldlm_lockd.c:2901:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 4297.414070] LustreError: 9123:0:(ldlm_lockd.c:2901:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 4300.789030] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 22:07:26 (1746151646)
[ 4346.814215] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 22:08:12 (1746151692)
[ 4402.933343] Lustre: 27886:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746151694/real 1746151694]  req@ffff8800a1c10a80 x1830968302545408/t0(0) o101->lustre-MDT0000-mdc-ffff8800a8696800@192.168.203.144@tcp:12/10 lens 576/1024 e 0 to 1 dl 1746151749 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'stat.0' uid:0 gid:0
[ 4402.946194] Lustre: 27886:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
[ 4402.970303] Lustre: lustre-MDT0000-mdc-ffff8800a8696800: Connection restored to  (at 192.168.203.144@tcp)
[ 4402.974565] Lustre: Skipped 30 previous similar messages
[ 4406.315110] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 22:09:12 (1746151752)
[ 4433.518464] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 22:09:39 (1746151779)
[ 4433.768702] Lustre: Mounted lustre-client
[ 4555.203056] LustreError: 29660:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 4555.209220] LustreError: 29660:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 4555.249600] Lustre: Unmounted lustre-client
[ 4558.376496] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 22:11:44 (1746151904)
[ 4558.734974] LustreError: 30334:0:(osc_request.c:2963:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 4000ms
[ 4560.834575] LustreError: lustre-OST0000-osc-ffff8800a8696800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4560.851827] LustreError: 30418:0:(ldlm_resource.c:982:ldlm_resource_complain()) lustre-OST0000-osc-ffff8800a8696800: namespace resource [0x240000400:0x80ad:0x0].0x0 (ffff88012a5ac900) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4560.863616] LustreError: 30418:0:(ldlm_resource.c:982:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4562.751823] LustreError: 30334:0:(osc_request.c:2963:osc_build_rpc()) cfs_fail_timeout id 414 awake
[ 4562.756589] Lustre: 1967:0:(llite_lib.c:4118:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.144@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 4565.578464] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 22:11:51 (1746151911)
[ 4625.892933] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 22:12:51 (1746151971)
[ 4626.715594] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 4627.729353] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 22:12:53 (1746151973)
[ 4683.653369] Lustre: lustre-MDT0000-mdc-ffff8800a8696800: Connection to lustre-MDT0000 (at 192.168.203.144@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4683.660759] Lustre: Skipped 17 previous similar messages
[ 4686.518044] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 4687.199179] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 22:13:53 (1746152033)
[ 4743.922502] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 22:14:49 (1746152089)
[ 4744.836475] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 4746.007666] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 22:14:51 (1746152091)
[ 4746.848427] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 4747.884078] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 22:14:53 (1746152093)
[ 4764.380285] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 22:15:10 (1746152110)
[ 4769.351586] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4788.042463] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 192.168.203.144@tcp) was lost; in progress operations using this service will fail
[ 4788.061200] LustreError: Skipped 2 previous similar messages
[ 4788.080443] Lustre: Evicted from MGS (at 192.168.203.144@tcp) after server handle changed from 0xed7431b5be5bc112 to 0xed7431b5be5bdae2
[ 4788.086995] Lustre: Skipped 2 previous similar messages
[ 4794.624196] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4795.425893] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4801.790282] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 22:15:47 (1746152147)
[ 4803.167814] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 4804.621691] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 22:15:50 (1746152150)
[ 4818.261470] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 22:16:04 (1746152164)
[ 4841.064003] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 22:16:27 (1746152187)
[ 4865.726524] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4866.491594] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4948.839292] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4949.598640] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4972.744622] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4973.625846] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5008.368023] Lustre: 1965:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746152300/real 1746152300]  req@ffff88012c519500 x1830968309105920/t0(0) o400->lustre-MDT0000-mdc-ffff8800a8696800@192.168.203.144@tcp:12/10 lens 224/224 e 0 to 1 dl 1746152355 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0
[ 5008.400002] Lustre: 1965:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
[ 5012.313906] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 22:19:18 (1746152358)
[ 5018.528031] LustreError: lustre-OST0000-osc-ffff8800a8696800: operation ost_setattr to node 192.168.203.144@tcp failed: rc = -19
[ 5018.567119] LustreError: Skipped 23 previous similar messages
[ 5044.044652] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5045.417951] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5209.357808] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 22:22:35 (1746152555)
[ 5293.964571] Lustre: lustre-MDT0000-mdc-ffff8800a8696800: Connection to lustre-MDT0000 (at 192.168.203.144@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5293.976952] Lustre: Skipped 8 previous similar messages
[ 5293.991077] Lustre: MGC192.168.203.144@tcp: Connection restored to 192.168.203.144@tcp (at 192.168.203.144@tcp)
[ 5293.996953] Lustre: Skipped 13 previous similar messages
[ 5316.893010] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 22:24:22 (1746152662)
[ 5317.774574] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 5318.691368] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 22:24:24 (1746152664)
[ 5319.826844] LustreError: lustre-MDT0000-mdc-ffff8800a8696800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5323.392633] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 22:24:29 (1746152669)
[ 5489.949642] Lustre: Evicted from lustre-OST0000_UUID (at 192.168.203.144@tcp) after server handle changed from 0xed7431b5be6b9aa3 to 0xed7431b5be6b9ba6
[ 5489.958637] Lustre: Skipped 5 previous similar messages
[ 5489.961057] LustreError: lustre-OST0000-osc-ffff8800a8696800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5490.820768] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 22:27:16 (1746152836)
[ 5527.899014] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 22:27:53 (1746152873)
[ 5528.807447] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 5529.815193] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 22:27:55 (1746152875)
[ 5530.623505] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 5531.617914] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 22:27:57 (1746152877)
[ 5532.881902] Lustre: DEBUG MARKER: SKIP: recovery-small test_152 MDS Linux kernel does not support killable semaphore
[ 5533.842830] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 22:27:59 (1746152879)
[ 5564.071684] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 192.168.203.144@tcp) was lost; in progress operations using this service will fail
[ 5564.090484] LustreError: Skipped 5 previous similar messages
[ 5571.298634] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 22:28:37 (1746152917)
[ 5571.974812] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 5572.881724] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 22:28:38 (1746152918)
[ 5573.535143] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 5574.557575] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 22:28:40 (1746152920)
[ 5578.884268] LustreError: 18364:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup
[ 5578.886812] LustreError: 18364:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 5578.949028] Lustre: Unmounted lustre-client
[ 5579.705412] Lustre: Mounted lustre-client
[ 5581.414796] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5600.220266] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: total 2264