[ 1632.345779] LustreError: lustre-OST0000-osc-ffff8800ada3f000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1632.359126] Lustre: 1982:0:(llite_lib.c:4117:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.127@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1635.142609] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 18:23:06 (1746138186)
[ 1635.998394] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1636.873060] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 18:23:08 (1746138188)
[ 1637.652876] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1638.456842] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 18:23:09 (1746138189)
[ 1658.836106] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 192.168.201.127@tcp) was lost; in progress operations using this service will fail
[ 1658.875232] Lustre: Evicted from MGS (at 192.168.201.127@tcp) after server handle changed from 0x344fb49230470ec4 to 0x344fb492304730cf
[ 1753.700839] LustreError: lustre-MDT0000-mdc-ffff8800ada3f000: operation mds_reint to node 192.168.201.127@tcp failed: rc = -19
[ 1753.722694] LustreError: Skipped 4 previous similar messages
[ 1771.003886] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 192.168.201.127@tcp) was lost; in progress operations using this service will fail
[ 1771.030504] Lustre: Evicted from MGS (at 192.168.201.127@tcp) after server handle changed from 0x344fb492304730cf to 0x344fb492304a4fcb
[ 1771.040031] Lustre: MGC192.168.201.127@tcp: Connection restored to 192.168.201.127@tcp (at 192.168.201.127@tcp)
[ 1771.060549] Lustre: Skipped 13 previous similar messages
[ 1784.587964] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 18:25:35 (1746138335)
[ 1784.900181] Lustre: *** cfs_fail_loc=305, val=0***
[ 1784.902639] Lustre: Skipped 4 previous similar messages
[ 1824.147991] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 192.168.201.127@tcp) was lost; in progress operations using this service will fail
[ 1824.172366] Lustre: lustre-MDT0000-mdc-ffff8800ada3f000: Connection to lustre-MDT0000 (at 192.168.201.127@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 1824.188037] Lustre: Skipped 13 previous similar messages
[ 1824.199938] Lustre: Evicted from MGS (at 192.168.201.127@tcp) after server handle changed from 0x344fb492304a4fcb to 0x344fb492304a54da
[ 1829.625143] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1830.937502] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1836.855253] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 18:26:27 (1746138387)
[ 1849.197218] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 192.168.201.127@tcp) was lost; in progress operations using this service will fail
[ 1849.221232] Lustre: Evicted from MGS (at 192.168.201.127@tcp) after server handle changed from 0x344fb492304a54da to 0x344fb492304a5b01
[ 1859.280187] LustreError: lustre-MDT0000-mdc-ffff8800ada3f000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 1870.175891] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 18:27:01 (1746138421)
[ 1880.129833] LustreError: lustre-OST0000-osc-ffff8800ada3f000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1893.255055] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 18:27:24 (1746138444)
[ 1905.245052] LustreError: lustre-MDT0000-mdc-ffff8800ada3f000: operation mds_reint to node 192.168.201.127@tcp failed: rc = -19
[ 1924.343560] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 192.168.201.127@tcp) was lost; in progress operations using this service will fail
[ 1924.377116] Lustre: Evicted from MGS (at 192.168.201.127@tcp) after server handle changed from 0x344fb492304a5b01 to 0x344fb492304ab870
[ 1929.058855] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1930.650440] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1993.970675] LustreError: lustre-MDT0000-mdc-ffff8800ada3f000: operation mds_reint to node 192.168.201.127@tcp failed: rc = -19
[ 2011.131616] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 192.168.201.127@tcp) was lost; in progress operations using this service will fail
[ 2011.174740] Lustre: Evicted from MGS (at 192.168.201.127@tcp) after server handle changed from 0x344fb492304ab870 to 0x344fb492304cde9f
[ 2020.130764] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2022.171851] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2104.351454] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 192.168.201.127@tcp) was lost; in progress operations using this service will fail
[ 2104.387446] Lustre: Evicted from MGS (at 192.168.201.127@tcp) after server handle changed from 0x344fb492304cde9f to 0x344fb492304f3239
[ 2112.211040] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2113.666830] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2140.402368] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 18:31:31 (1746138691)
[ 2144.371859] LustreError: lustre-MDT0000-mdc-ffff8800ada3f000: operation mds_close to node 192.168.201.127@tcp failed: rc = -19
[ 2144.386800] LustreError: Skipped 3 previous similar messages
[ 2165.353877] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2184.462184] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 192.168.201.127@tcp) was lost; in progress operations using this service will fail
[ 2184.484472] LustreError: Skipped 1 previous similar message
[ 2184.517270] Lustre: Evicted from MGS (at 192.168.201.127@tcp) after server handle changed from 0x344fb492305031c7 to 0x344fb49230504ff0
[ 2184.533253] Lustre: Skipped 1 previous similar message
[ 2189.667280] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2220.779184] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2254.685205] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2296.866985] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2339.670282] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 192.168.201.127@tcp) was lost; in progress operations using this service will fail
[ 2339.681229] LustreError: Skipped 3 previous similar messages
[ 2339.714439] Lustre: Evicted from MGS (at 192.168.201.127@tcp) after server handle changed from 0x344fb492305208e2 to 0x344fb492305302a4
[ 2339.718020] Lustre: Skipped 3 previous similar messages
[ 2342.521144] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2389.756803] Lustre: MGC192.168.201.127@tcp: Connection restored to 192.168.201.127@tcp (at 192.168.201.127@tcp)
[ 2389.764227] Lustre: Skipped 24 previous similar messages
[ 2416.753878] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 18:36:08 (1746138968)
[ 2428.341673] LustreError: lustre-OST0000-osc-ffff8800ada3f000: operation ldlm_enqueue to node 192.168.201.127@tcp failed: rc = -19
[ 2428.351303] LustreError: Skipped 7 previous similar messages
[ 2428.355379] Lustre: lustre-OST0000-osc-ffff8800ada3f000: Connection to lustre-OST0000 (at 192.168.201.127@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2428.365370] Lustre: Skipped 12 previous similar messages
[ 2454.324281] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2455.777913] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 2786.939483] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2788.700307] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3092.905184] LustreError: lustre-OST0000-osc-ffff8800ada3f000: operation ldlm_enqueue to node 192.168.201.127@tcp failed: rc = -19
[ 3092.911353] LustreError: Skipped 1 previous similar message
[ 3092.924070] Lustre: lustre-OST0000-osc-ffff8800ada3f000: Connection to lustre-OST0000 (at 192.168.201.127@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3092.969697] Lustre: Skipped 1 previous similar message
[ 3120.613850] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3121.863061] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3385.810861] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 18:52:16 (1746139936)
[ 3402.774685] Lustre: 22524:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746139938/real 1746139938]  req@ffff88012d58a680 x1830956977268480/t0(0) o101->lustre-MDT0000-mdc-ffff8800ada3f000@192.168.201.127@tcp:12/10 lens 576/1024 e 0 to 1 dl 1746139954 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0
[ 3402.796932] Lustre: 22524:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 10 previous similar messages
[ 3402.828009] Lustre: lustre-MDT0000-mdc-ffff8800ada3f000: Connection restored to 192.168.201.127@tcp (at 192.168.201.127@tcp)
[ 3402.833336] Lustre: Skipped 2 previous similar messages
[ 3407.516627] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 18:52:38 (1746139958)
[ 3429.258411] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 18:53:00 (1746139980)
[ 3451.337982] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 18:53:22 (1746140002)
[ 3451.572264] Lustre: Mounted lustre-client
[ 3481.614834] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 192.168.201.127@tcp) was lost; in progress operations using this service will fail
[ 3481.624000] LustreError: Skipped 1 previous similar message
[ 3481.647770] Lustre: Evicted from MGS (at 192.168.201.127@tcp) after server handle changed from 0x344fb49230545d7f to 0x344fb4923073537e
[ 3481.663875] Lustre: Skipped 1 previous similar message
[ 3482.631745] Lustre: 1983:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746140018/real 1746140018]  req@ffff8800a7ca0380 x1830956977284992/t0(0) o400->lustre-MDT0000-mdc-ffff8800ada3f000@192.168.201.127@tcp:12/10 lens 224/224 e 0 to 1 dl 1746140034 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0
[ 3482.670425] Lustre: 1983:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 3485.995711] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3487.246309] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3489.150620] LustreError: 25536:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 3489.182357] LustreError: 25536:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 3489.199469] Lustre: Unmounted lustre-client
[ 3493.501555] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 18:54:04 (1746140044)
[ 3632.837498] Lustre: 1981:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746140168/real 1746140168]  req@ffff88012e22ea00 x1830956977331840/t0(0) o4->lustre-OST0000-osc-ffff8800ada3f000@192.168.201.127@tcp:6/4 lens 488/448 e 0 to 1 dl 1746140184 ref 2 fl Rpc:XQr/202/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0
[ 3632.861019] Lustre: 1981:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 60 previous similar messages
[ 3696.911487] Lustre: lustre-OST0000-osc-ffff8800ada3f000: Connection to lustre-OST0000 (at 192.168.201.127@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3696.938955] Lustre: Skipped 16 previous similar messages
[ 3793.163025] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 18:59:04 (1746140344)
[ 3838.786961] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 18:59:49 (1746140389)
[ 3840.708540] LustreError: 28105:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup
[ 3840.711470] LustreError: 28105:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 3840.749995] Lustre: Unmounted lustre-client
[ 3854.909627] Lustre: Mounted lustre-client
[ 3858.873739] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 19:00:09 (1746140409)
[ 3859.092458] LustreError: 29163:0:(mdc_locks.c:1333:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 3860.204440] LustreError: 29163:0:(mdc_locks.c:1333:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 3860.282183] Lustre: *** cfs_fail_loc=305, val=0***
[ 3879.665828] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 19:00:30 (1746140430)
[ 3879.964930] Lustre: Mounted lustre-client
[ 3880.743045] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 3890.771636] LustreError: 29905:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 3890.777207] LustreError: 29905:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages
[ 3890.794493] Lustre: Unmounted lustre-client
[ 3894.284755] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 19:00:45 (1746140445)
[ 3954.545820] LustreError: lustre-MDT0000-mdc-ffff88012e2df000: operation mds_reint to node 192.168.201.127@tcp failed: rc = -19
[ 3970.108634] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 192.168.201.127@tcp) was lost; in progress operations using this service will fail
[ 3970.148519] Lustre: Evicted from MGS (at 192.168.201.127@tcp) after server handle changed from 0x344fb49230735fe1 to 0x344fb492307617ab
[ 4085.642354] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 19:03:56 (1746140636)
[ 4089.148009] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4100.393066] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 192.168.201.127@tcp) was lost; in progress operations using this service will fail
[ 4100.428326] Lustre: Evicted from MGS (at 192.168.201.127@tcp) after server handle changed from 0x344fb492307617ab to 0x344fb492307adbf7
[ 4100.453228] Lustre: MGC192.168.201.127@tcp: Connection restored to 192.168.201.127@tcp (at 192.168.201.127@tcp)
[ 4100.457018] Lustre: Skipped 25 previous similar messages
[ 4100.468449] LustreError: lustre-MDT0000-mdc-ffff88012e2df000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4108.517784] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 19:04:19 (1746140659)
[ 4108.986016] Lustre: Mounted lustre-client
[ 4116.105589] LustreError: lustre-OST0000-osc-ffff88012e2df000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4126.190519] Lustre: 1982:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746140662/real 1746140662]  req@ffff8800a20d4e00 x1830956982006784/t0(0) o103->lustre-OST0000-osc-ffff8800aa58d800@192.168.201.127@tcp:17/18 lens 328/224 e 0 to 1 dl 1746140678 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0
[ 4126.217727] Lustre: 1982:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 123 previous similar messages
[ 4126.969725] LustreError: 848:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 4126.973068] LustreError: 848:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 4127.016559] Lustre: Unmounted lustre-client
[ 4131.353224] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 19:04:42 (1746140682)
[ 4137.537492] LustreError: lustre-MDT0000-mdc-ffff88012e2df000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4137.545874] LustreError: 1563:0:(file.c:6135:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4142.646955] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 19:04:53 (1746140693)
[ 4142.749106] LustreError: 2344:0:(recover.c:322:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4147.760559] LustreError: 2344:0:(recover.c:322:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=0
[ 4147.775527] LustreError: 2344:0:(import.c:703:ptlrpc_connect_import_locked()) already connected
[ 4147.923812] LustreError: lustre-MDT0000-mdc-ffff88012e2df000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4147.937072] LustreError: 2369:0:(import.c:287:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4147.983016] LustreError: 2374:0:(file.c:6135:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4149.038217] LustreError: 2381:0:(file.c:6135:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4149.047685] LustreError: 2381:0:(file.c:6135:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4151.276565] LustreError: 2405:0:(file.c:6135:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4151.284498] LustreError: 2405:0:(file.c:6135:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4155.618526] LustreError: 2452:0:(file.c:6135:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4155.631689] LustreError: 2452:0:(file.c:6135:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4163.013588] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 19:05:14 (1746140714)
[ 4188.659830] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4189.731973] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4197.352241] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 19:05:48 (1746140748)
[ 4222.892855] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4223.968976] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4231.146545] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 19:06:22 (1746140782)
[ 4283.677103] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4285.043409] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4291.576071] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 19:07:22 (1746140842)
[ 4298.570659] Lustre: lustre-OST0000-osc-ffff88012e2df000: Connection to lustre-OST0000 (at 192.168.201.127@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4298.581557] Lustre: Skipped 18 previous similar messages
[ 4317.878134] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4319.202539] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4323.157759] LustreError: 8518:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup
[ 4323.164074] LustreError: 8518:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 4323.209679] Lustre: Unmounted lustre-client
[ 4357.985723] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4359.956250] Lustre: Mounted lustre-client
[ 4364.033051] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 19:08:35 (1746140915)
[ 4365.687070] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 4366.672313] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 19:08:37 (1746140917)
[ 4386.822219] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 19:08:57 (1746140937)
[ 4387.943327] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 4389.047993] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 19:09:00 (1746140940)
[ 4390.139561] Lustre: *** cfs_fail_loc=805, val=0***
[ 4390.183946] Lustre: Mounted lustre-client
[ 4393.806317] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4410.208245] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 192.168.201.127@tcp) was lost; in progress operations using this service will fail
[ 4410.233416] Lustre: Evicted from MGS (at 192.168.201.127@tcp) after server handle changed from 0x344fb492307ae5f2 to 0x344fb492307ae8f4
[ 4414.497117] LustreError: 12068:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 4414.500831] LustreError: 12068:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages
[ 4414.527154] Lustre: Unmounted lustre-client
[ 4419.393490] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 19:09:30 (1746140970)
[ 4446.223337] LustreError: 1979:0:(client.c:3262:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff8800a7c26a00 x1830956982081408/t94489280516(94489280516) o101->lustre-MDT0000-mdc-ffff8800b616b000@192.168.201.127@tcp:12/10 lens 664/608 e 0 to 0 dl 1746141014 ref 2 fl Interpret:RPQU/204/0 rc 301/301 job:'multiop.0' uid:0 gid:0
[ 4447.845988] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4448.530458] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4450.056612] checkstat (13578) used greatest stack depth: 10672 bytes left
[ 4453.149977] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 19:10:04 (1746141004)
[ 4453.879416] LustreError: lustre-OST0000-osc-ffff8800b616b000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4453.891091] Lustre: 1982:0:(llite_lib.c:4117:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.127@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 4458.536830] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 19:10:09 (1746141009)
[ 4459.227503] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 4460.266311] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 19:10:11 (1746141011)
[ 4461.105742] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 4462.131809] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 19:10:13 (1746141013)
[ 4462.908007] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 4463.902426] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 19:10:15 (1746141015)
[ 4464.829016] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 4465.822595] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 19:10:17 (1746141017)
[ 4466.583142] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 4467.448131] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 19:10:18 (1746141018)
[ 4468.298202] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 4469.504917] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 19:10:20 (1746141020)
[ 4470.502199] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 4471.432690] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 19:10:22 (1746141022)
[ 4472.152274] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 4473.060068] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 19:10:24 (1746141024)
[ 4473.864160] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 4474.857192] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 19:10:26 (1746141026)
[ 4475.761948] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 4476.704734] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 19:10:27 (1746141027)
[ 4477.416594] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 4478.350516] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 19:10:29 (1746141029)
[ 4479.591364] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 4480.516597] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 19:10:31 (1746141031)
[ 4497.987130] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 19:10:49 (1746141049)
[ 4524.586709] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 19:11:15 (1746141075)
[ 4525.124866] Lustre: *** cfs_fail_loc=51b, val=3***
[ 4532.432069] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 19:11:23 (1746141083)
[ 4532.677174] Lustre: *** cfs_fail_loc=51b, val=4***
[ 4540.433946] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 19:11:31 (1746141091)
[ 4540.645598] Lustre: *** cfs_fail_loc=50f, val=3***
[ 4545.476930] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 19:11:36 (1746141096)
[ 4545.658695] Lustre: *** cfs_fail_loc=50f, val=4***
[ 4551.831043] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 19:11:42 (1746141102)
[ 4552.051793] Lustre: *** cfs_fail_loc=510, val=3***
[ 4558.071994] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 19:11:49 (1746141109)
[ 4558.329911] Lustre: *** cfs_fail_loc=51b, val=3***
[ 4566.211177] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 19:11:57 (1746141117)
[ 4566.670646] Lustre: *** cfs_fail_loc=51c, val=3***
[ 4630.021773] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 19:13:00 (1746141180)
[ 4630.355964] LustreError: 25424:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 4633.004766] LustreError: lustre-MDT0000-mdc-ffff8800b616b000: operation ldlm_enqueue to node 192.168.201.127@tcp failed: rc = -107
[ 4633.010086] LustreError: Skipped 4 previous similar messages
[ 4633.019899] LustreError: lustre-MDT0000-mdc-ffff8800b616b000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4633.031371] LustreError: 25440:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800b616b000: inode [0x20000a042:0xb:0x0] mdc close failed: rc = -108
[ 4633.055582] LustreError: 25440:0:(ldlm_resource.c:982:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8800b616b000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8800b6d62e00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4633.069789] LustreError: 25445:0:(file.c:6135:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4633.069792] LustreError: 25445:0:(file.c:6135:ll_inode_revalidate_fini()) Skipped 20 previous similar messages
[ 4634.379456] LustreError: 25424:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 4636.480105] LustreError: 25469:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 4639.115104] LustreError: lustre-MDT0000-mdc-ffff8800b616b000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4639.135103] LustreError: 25485:0:(ldlm_resource.c:982:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8800b616b000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8800b6d62d00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4639.161022] LustreError: 25485:0:(ldlm_resource.c:982:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4640.502818] LustreError: 25469:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 4647.152889] LustreError: lustre-MDT0000-mdc-ffff8800b616b000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4647.482147] LustreError: 25532:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 4650.145465] LustreError: lustre-MDT0000-mdc-ffff8800b616b000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4650.226051] LustreError: 25553:0:(file.c:6135:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4650.231928] LustreError: 25553:0:(file.c:6135:ll_inode_revalidate_fini()) Skipped 2 previous similar messages
[ 4651.497519] LustreError: 25532:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 4651.508814] LustreError: 25532:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800b616b000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 4651.518450] LustreError: 25532:0:(file.c:247:ll_close_inode_openhandle()) Skipped 3 previous similar messages
[ 4656.987429] LustreError: 25613:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 4659.547485] LustreError: lustre-MDT0000-mdc-ffff8800b616b000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4659.566222] LustreError: 25629:0:(ldlm_resource.c:982:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8800b616b000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8800b6d62f00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4659.585237] LustreError: 25629:0:(ldlm_resource.c:982:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4661.002077] LustreError: 25613:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 4667.492222] LustreError: lustre-MDT0000-mdc-ffff8800b616b000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4667.724343] LustreError: 25675:0:(ldlm_flock.c:856:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 4670.306209] LustreError: lustre-MDT0000-mdc-ffff8800b616b000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4670.316921] LustreError: 25691:0:(ldlm_resource.c:982:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8800b616b000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8800b6d62d00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4670.326064] LustreError: 25691:0:(ldlm_resource.c:982:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4671.739516] LustreError: 25675:0:(ldlm_flock.c:856:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 4674.294815] LustreError: lustre-MDT0000-mdc-ffff8800b616b000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4675.829813] LustreError: 25705:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800b616b000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 4682.847113] LustreError: 25781:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 4682.857637] LustreError: 25781:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 4685.399668] LustreError: lustre-MDT0000-mdc-ffff8800b616b000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4685.445226] LustreError: 25807:0:(file.c:6135:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4685.451628] LustreError: 25807:0:(file.c:6135:ll_inode_revalidate_fini()) Skipped 53 previous similar messages
[ 4686.890492] LustreError: 25781:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 4686.896749] LustreError: 25781:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 4686.908204] LustreError: 25781:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800b616b000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 4693.589626] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 19:14:04 (1746141244)
[ 4710.659337] Lustre: lustre-MDT0000-mdc-ffff8800b616b000: Connection restored to 192.168.201.127@tcp (at 192.168.201.127@tcp)
[ 4710.664109] Lustre: Skipped 26 previous similar messages
[ 4711.484196] LustreError: 9383:0:(ldlm_lockd.c:2901:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 4711.494128] LustreError: 9383:0:(ldlm_lockd.c:2901:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 4715.524441] LustreError: 9383:0:(ldlm_lockd.c:2901:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 4715.528410] LustreError: 9383:0:(ldlm_lockd.c:2901:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 4720.059907] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 19:14:31 (1746141271)
[ 4762.240744] Lustre: 27366:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746141273/real 1746141273]  req@ffff8800a20d5f80 x1830956982198144/t0(0) o101->lustre-MDT0000-mdc-ffff8800b616b000@192.168.201.127@tcp:12/10 lens 576/1024 e 1 to 1 dl 1746141314 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'stat.0' uid:0 gid:0
[ 4762.256536] Lustre: 27366:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
[ 4765.601286] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 19:15:16 (1746141316)
[ 4825.090150] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 19:16:16 (1746141376)
[ 4853.583492] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 19:16:44 (1746141404)
[ 4853.810904] Lustre: Mounted lustre-client
[ 4975.491655] LustreError: 29960:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 4975.499561] LustreError: 29960:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 4975.555113] Lustre: Unmounted lustre-client
[ 4980.324876] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 19:18:51 (1746141531)
[ 4980.736561] LustreError: 30634:0:(osc_request.c:2961:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 4000ms
[ 4983.444844] Lustre: lustre-OST0000-osc-ffff8800b616b000: Connection to lustre-OST0000 (at 192.168.201.127@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4983.458212] Lustre: Skipped 19 previous similar messages
[ 4983.470104] LustreError: lustre-OST0000-osc-ffff8800b616b000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4983.481893] LustreError: 30724:0:(ldlm_resource.c:982:ldlm_resource_complain()) lustre-OST0000-osc-ffff8800b616b000: namespace resource [0x240000400:0x570d:0x0].0x0 (ffff8800b6e40200) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4983.503933] LustreError: 30724:0:(ldlm_resource.c:982:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4984.758640] LustreError: 30634:0:(osc_request.c:2961:osc_build_rpc()) cfs_fail_timeout id 414 awake
[ 4984.765337] Lustre: 1983:0:(llite_lib.c:4117:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.127@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 4988.496178] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 19:18:59 (1746141539)
[ 5049.853935] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 19:20:00 (1746141600)
[ 5050.709334] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 5051.874691] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 19:20:02 (1746141602)
[ 5111.428650] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 5112.375429] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 19:21:03 (1746141663)
[ 5169.449277] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 19:22:00 (1746141720)
[ 5170.440096] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 5171.535062] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 19:22:02 (1746141722)
[ 5172.582993] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 5173.728737] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 19:22:04 (1746141724)
[ 5194.147956] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 19:22:25 (1746141745)
[ 5200.752177] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5223.709834] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 192.168.201.127@tcp) was lost; in progress operations using this service will fail
[ 5223.723559] LustreError: Skipped 2 previous similar messages
[ 5223.739377] Lustre: Evicted from MGS (at 192.168.201.127@tcp) after server handle changed from 0x344fb492307af3e4 to 0x344fb492307b0e47
[ 5223.751842] Lustre: Skipped 2 previous similar messages
[ 5226.990675] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5228.000739] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5235.039069] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 19:23:06 (1746141786)
[ 5236.660892] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 5237.948359] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 19:23:08 (1746141788)
[ 5253.856762] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 19:23:25 (1746141805)
[ 5278.181023] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 19:23:49 (1746141829)
[ 5281.505404] LustreError: lustre-OST0000-osc-ffff8800b616b000: operation ost_setattr to node 192.168.201.127@tcp failed: rc = -107
[ 5281.513632] LustreError: Skipped 9 previous similar messages
[ 5305.372833] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5306.431497] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5389.003524] Lustre: MGC192.168.201.127@tcp: Connection restored to 192.168.201.127@tcp (at 192.168.201.127@tcp)
[ 5389.010921] Lustre: Skipped 12 previous similar messages
[ 5390.797043] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5391.680362] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5414.159790] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5414.995966] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5429.006542] Lustre: 1980:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746141925/real 1746141925]  req@ffff88009e149c00 x1830956988760192/t0(0) o400->lustre-MDT0000-mdc-ffff8800b616b000@192.168.201.127@tcp:12/10 lens 224/224 e 0 to 1 dl 1746141980 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0
[ 5429.020512] Lustre: 1980:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 10 previous similar messages
[ 5451.166455] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 19:26:42 (1746142002)
[ 5482.380476] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5484.726230] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5652.450800] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 19:30:03 (1746142203)
[ 5749.659317] Lustre: lustre-MDT0000-mdc-ffff8800b616b000: Connection to lustre-MDT0000 (at 192.168.201.127@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5749.682179] Lustre: Skipped 10 previous similar messages
[ 5749.685585] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 192.168.201.127@tcp) was lost; in progress operations using this service will fail
[ 5749.693155] LustreError: Skipped 4 previous similar messages
[ 5749.714400] Lustre: Evicted from MGS (at 192.168.201.127@tcp) after server handle changed from 0x344fb492307b25bd to 0x344fb492308ac819
[ 5749.732187] Lustre: Skipped 4 previous similar messages
[ 5772.115879] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 19:32:03 (1746142323)
[ 5772.909446] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 5773.917099] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 19:32:05 (1746142325)
[ 5775.281321] LustreError: lustre-MDT0000-mdc-ffff8800b616b000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5780.176514] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 19:32:11 (1746142331)
[ 5946.323450] LustreError: lustre-OST0000-osc-ffff8800b616b000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5947.403478] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 19:34:58 (1746142498)
[ 5984.073341] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 19:35:35 (1746142535)
[ 5984.833261] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 5985.860402] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 19:35:36 (1746142536)
[ 5986.794812] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 5988.023081] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 19:35:39 (1746142539)
[ 5989.757643] Lustre: DEBUG MARKER: SKIP: recovery-small test_152 MDS Linux kernel does not support killable semaphore
[ 5990.722594] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 19:35:41 (1746142541)
[ 6021.440745] Lustre: MGC192.168.201.127@tcp: Connection restored to 192.168.201.127@tcp (at 192.168.201.127@tcp)
[ 6021.455873] Lustre: Skipped 7 previous similar messages
[ 6030.670864] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 19:36:21 (1746142581)
[ 6031.754111] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 6032.908328] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 19:36:23 (1746142583)
[ 6033.841164] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 6035.205246] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 19:36:26 (1746142586)
[ 6040.668704] LustreError: 18700:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup
[ 6040.688403] LustreError: 18700:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 6040.804635] Lustre: Unmounted lustre-client
[ 6041.786909] Lustre: Mounted lustre-client
[ 6043.635479] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6065.190308] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: total 2260