[ 1443.860290] LustreError: lustre-OST0000-osc-ffff8800b38fa000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1443.869473] Lustre: 1964:0:(llite_lib.c:4118:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.129@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1446.620296] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 22:17:27 (1746065847)
[ 1447.567527] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1448.401180] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 22:17:29 (1746065849)
[ 1449.210617] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1449.966468] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 22:17:30 (1746065850)
[ 1468.703967] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 192.168.204.129@tcp) was lost; in progress operations using this service will fail
[ 1468.727280] Lustre: Evicted from MGS (at 192.168.204.129@tcp) after server handle changed from 0x10b27947f1aad0f to 0x10b27947f1acda0
[ 1559.900201] LustreError: lustre-MDT0000-mdc-ffff8800b38fa000: operation ldlm_enqueue to node 192.168.204.129@tcp failed: rc = -19
[ 1559.908012] LustreError: Skipped 3 previous similar messages
[ 1573.915924] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 192.168.204.129@tcp) was lost; in progress operations using this service will fail
[ 1573.935768] Lustre: Evicted from MGS (at 192.168.204.129@tcp) after server handle changed from 0x10b27947f1acda0 to 0x10b27947f202a87
[ 1582.296109] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 22:19:43 (1746065983)
[ 1582.504850] Lustre: *** cfs_fail_loc=305, val=0***
[ 1582.506639] Lustre: Skipped 4 previous similar messages
[ 1619.018801] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 192.168.204.129@tcp) was lost; in progress operations using this service will fail
[ 1619.033498] Lustre: Evicted from MGS (at 192.168.204.129@tcp) after server handle changed from 0x10b27947f202a87 to 0x10b27947f202f34
[ 1621.024768] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1621.945819] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1626.307627] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 22:20:27 (1746066027)
[ 1634.042043] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 192.168.204.129@tcp) was lost; in progress operations using this service will fail
[ 1634.063474] Lustre: Evicted from MGS (at 192.168.204.129@tcp) after server handle changed from 0x10b27947f202f34 to 0x10b27947f2033b0
[ 1634.086782] LustreError: lustre-MDT0000-mdc-ffff8800b38fa000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 1642.434066] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 22:20:43 (1746066043)
[ 1649.642146] LustreError: lustre-OST0000-osc-ffff8800b38fa000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1658.295141] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 22:20:59 (1746066059)
[ 1669.243144] LustreError: lustre-MDT0000-mdc-ffff8800b38fa000: operation ldlm_enqueue to node 192.168.204.129@tcp failed: rc = -19
[ 1684.141511] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 192.168.204.129@tcp) was lost; in progress operations using this service will fail
[ 1684.162549] Lustre: Evicted from MGS (at 192.168.204.129@tcp) after server handle changed from 0x10b27947f2033b0 to 0x10b27947f20b9d6
[ 1691.165990] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1692.012121] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1769.399734] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 192.168.204.129@tcp) was lost; in progress operations using this service will fail
[ 1769.415755] Lustre: Evicted from MGS (at 192.168.204.129@tcp) after server handle changed from 0x10b27947f20b9d6 to 0x10b27947f23cc76
[ 1776.478834] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1777.436319] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1840.196024] LustreError: lustre-MDT0000-mdc-ffff8800b38fa000: operation mds_reint to node 192.168.204.129@tcp failed: rc = -19
[ 1840.202686] LustreError: Skipped 1 previous similar message
[ 1854.585131] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 192.168.204.129@tcp) was lost; in progress operations using this service will fail
[ 1854.594556] Lustre: Evicted from MGS (at 192.168.204.129@tcp) after server handle changed from 0x10b27947f23cc76 to 0x10b27947f27357e
[ 1861.342020] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1862.271795] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1886.679932] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 22:24:47 (1746066287)
[ 1906.282854] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 1924.677146] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 192.168.204.129@tcp) was lost; in progress operations using this service will fail
[ 1924.683785] LustreError: Skipped 1 previous similar message
[ 1924.715311] Lustre: Evicted from MGS (at 192.168.204.129@tcp) after server handle changed from 0x10b27947f2898ef to 0x10b27947f28ce10
[ 1924.721963] Lustre: Skipped 1 previous similar message
[ 1924.724980] Lustre: MGC192.168.204.129@tcp: Connection restored to 192.168.204.129@tcp (at 192.168.204.129@tcp)
[ 1924.728095] Lustre: Skipped 24 previous similar messages
[ 1925.896122] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 1927.419684] hrtimer: interrupt took 4456446 ns
[ 1932.095741] Lustre: lustre-MDT0000-mdc-ffff8800b38fa000: Connection to lustre-MDT0000 (at 192.168.204.129@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 1932.110473] Lustre: Skipped 17 previous similar messages
[ 1949.164096] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 1977.605321] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2016.098648] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2054.888369] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 192.168.204.129@tcp) was lost; in progress operations using this service will fail
[ 2054.893108] LustreError: Skipped 3 previous similar messages
[ 2054.900145] Lustre: Evicted from MGS (at 192.168.204.129@tcp) after server handle changed from 0x10b27947f2abe1b to 0x10b27947f2c2c0c
[ 2054.904327] Lustre: Skipped 3 previous similar messages
[ 2057.454435] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2127.683336] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 22:28:48 (1746066528)
[ 2138.771472] LustreError: lustre-OST0000-osc-ffff8800b38fa000: operation ost_write to node 192.168.204.129@tcp failed: rc = -19
[ 2138.779012] LustreError: Skipped 9 previous similar messages
[ 2160.934268] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2161.632554] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 2493.989235] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2494.757707] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 2800.734742] LustreError: lustre-OST0000-osc-ffff8800b38fa000: operation ldlm_enqueue to node 192.168.204.129@tcp failed: rc = -19
[ 2800.738217] LustreError: Skipped 1 previous similar message
[ 2800.740481] Lustre: lustre-OST0000-osc-ffff8800b38fa000: Connection to lustre-OST0000 (at 192.168.204.129@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2800.746119] Lustre: Skipped 6 previous similar messages
[ 2818.158210] Lustre: lustre-OST0000-osc-ffff8800b38fa000: Connection restored to 192.168.204.129@tcp (at 192.168.204.129@tcp)
[ 2818.170064] Lustre: Skipped 12 previous similar messages
[ 2821.990422] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2822.724799] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3093.417180] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 22:44:54 (1746067494)
[ 3110.162818] Lustre: 22336:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746067495/real 1746067495]  req@ffff88012bf60380 x1830881322047232/t0(0) o101->lustre-MDT0000-mdc-ffff8800b38fa000@192.168.204.129@tcp:12/10 lens 576/1024 e 0 to 1 dl 1746067511 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0
[ 3110.174481] Lustre: 22336:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
[ 3112.818458] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 22:45:13 (1746067513)
[ 3132.468200] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 22:45:33 (1746067533)
[ 3152.327983] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 22:45:53 (1746067553)
[ 3152.654204] Lustre: Mounted lustre-client
[ 3177.692452] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 192.168.204.129@tcp) was lost; in progress operations using this service will fail
[ 3177.697049] LustreError: Skipped 1 previous similar message
[ 3177.710124] Lustre: Evicted from MGS (at 192.168.204.129@tcp) after server handle changed from 0x10b27947f2dda38 to 0x10b27947f62a10a
[ 3177.719453] Lustre: Skipped 1 previous similar message
[ 3183.073500] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3183.823484] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3185.333938] LustreError: 25338:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 3185.341322] LustreError: 25338:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 3185.356718] Lustre: Unmounted lustre-client
[ 3188.072410] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 22:46:29 (1746067589)
[ 3188.700920] Lustre: 1966:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746067574/real 1746067574]  req@ffff8800a8614e00 x1830881322066432/t0(0) o400->lustre-MDT0000-mdc-ffff8800b38fa000@192.168.204.129@tcp:12/10 lens 224/224 e 0 to 1 dl 1746067590 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0
[ 3188.711129] Lustre: 1966:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
[ 3338.783810] Lustre: 1963:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746067724/real 1746067724]  req@ffff88012bf63100 x1830881322126208/t0(0) o4->lustre-OST0000-osc-ffff8800b38fa000@192.168.204.129@tcp:6/4 lens 488/448 e 0 to 1 dl 1746067740 ref 2 fl Rpc:XQr/202/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0
[ 3338.800148] Lustre: 1963:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 61 previous similar messages
[ 3402.848975] Lustre: lustre-OST0000-osc-ffff8800b38fa000: Connection to lustre-OST0000 (at 192.168.204.129@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3402.855379] Lustre: Skipped 17 previous similar messages
[ 3418.877782] Lustre: lustre-OST0000-osc-ffff8800b38fa000: Connection restored to 192.168.204.129@tcp (at 192.168.204.129@tcp)
[ 3418.884262] Lustre: Skipped 18 previous similar messages
[ 3476.821431] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 22:51:17 (1746067877)
[ 3520.635293] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 22:52:01 (1746067921)
[ 3521.994818] LustreError: 27889:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup
[ 3521.997536] LustreError: 27889:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 3522.021357] Lustre: Unmounted lustre-client
[ 3531.745421] Lustre: Mounted lustre-client
[ 3534.335168] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 22:52:15 (1746067935)
[ 3534.495968] LustreError: 28940:0:(mdc_locks.c:1333:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 3535.503706] LustreError: 28940:0:(mdc_locks.c:1333:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 3535.566364] Lustre: *** cfs_fail_loc=305, val=0***
[ 3554.186183] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 22:52:35 (1746067955)
[ 3554.324467] Lustre: Mounted lustre-client
[ 3554.770866] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 3564.798511] LustreError: 29683:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 3564.802140] LustreError: 29683:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages
[ 3564.817120] Lustre: Unmounted lustre-client
[ 3567.285107] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 22:52:48 (1746067968)
[ 3609.008711] LustreError: lustre-MDT0000-mdc-ffff88012b0fb000: operation mds_reint to node 192.168.204.129@tcp failed: rc = -19
[ 3624.438812] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 192.168.204.129@tcp) was lost; in progress operations using this service will fail
[ 3624.449931] Lustre: Evicted from MGS (at 192.168.204.129@tcp) after server handle changed from 0x10b27947f62b076 to 0x10b27947f658447
[ 3688.084613] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 22:54:49 (1746068089)
[ 3690.439069] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 3699.576641] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 192.168.204.129@tcp) was lost; in progress operations using this service will fail
[ 3699.585122] Lustre: Evicted from MGS (at 192.168.204.129@tcp) after server handle changed from 0x10b27947f658447 to 0x10b27947f6a180a
[ 3699.622163] LustreError: lustre-MDT0000-mdc-ffff88012b0fb000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 3704.044428] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 22:55:05 (1746068105)
[ 3704.215165] Lustre: Mounted lustre-client
[ 3710.883233] LustreError: lustre-OST0000-osc-ffff88012b0fb000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 3710.897627] LustreError: 503:0:(ldlm_resource.c:982:ldlm_resource_complain()) lustre-OST0000-osc-ffff88012b0fb000: namespace resource [0x240000400:0x8543:0x0].0x0 (ffff88009c880800) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 3720.830293] Lustre: 1966:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746068106/real 1746068106]  req@ffff88009eff8000 x1830881326759040/t0(0) o103->lustre-OST0000-osc-ffff88012c467000@192.168.204.129@tcp:17/18 lens 328/224 e 0 to 1 dl 1746068122 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0
[ 3720.844847] Lustre: 1966:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 78 previous similar messages
[ 3721.111562] LustreError: 538:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 3721.115284] LustreError: 538:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 3721.143164] Lustre: Unmounted lustre-client
[ 3723.472627] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 22:55:24 (1746068124)
[ 3728.569486] LustreError: lustre-MDT0000-mdc-ffff88012b0fb000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 3728.575339] LustreError: 1274:0:(file.c:6136:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 3731.556052] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 22:55:32 (1746068132)
[ 3731.606046] LustreError: 2047:0:(recover.c:322:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 3736.609270] LustreError: 2047:0:(recover.c:322:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=0
[ 3736.615037] LustreError: 2047:0:(import.c:703:ptlrpc_connect_import_locked()) already connected
[ 3737.338133] LustreError: lustre-MDT0000-mdc-ffff88012b0fb000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 3737.344778] LustreError: 2077:0:(import.c:287:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 3737.376025] LustreError: 2082:0:(file.c:6136:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3738.422343] LustreError: 2089:0:(file.c:6136:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3738.429987] LustreError: 2089:0:(file.c:6136:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 3740.553230] LustreError: 2112:0:(file.c:6136:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3740.559855] LustreError: 2112:0:(file.c:6136:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 3744.801883] LustreError: 2160:0:(file.c:6136:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3744.809450] LustreError: 2160:0:(file.c:6136:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 3750.357551] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 22:55:51 (1746068151)
[ 3771.013342] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3771.590466] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3776.828456] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 22:56:17 (1746068177)
[ 3796.924550] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3797.760539] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3803.004352] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 22:56:43 (1746068203)
[ 3850.189397] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3850.736374] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3854.446522] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 22:57:35 (1746068255)
[ 3874.001492] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3874.628624] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3877.171071] LustreError: 8198:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup
[ 3877.177283] LustreError: 8198:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 3877.209252] Lustre: Unmounted lustre-client
[ 3901.986511] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3903.532813] Lustre: Mounted lustre-client
[ 3905.944248] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 22:58:26 (1746068306)
[ 3907.065953] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 3907.697824] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 22:58:28 (1746068308)
[ 3918.101567] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 22:58:39 (1746068319)
[ 3918.746045] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 3919.532947] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 22:58:40 (1746068320)
[ 3920.128252] Lustre: *** cfs_fail_loc=805, val=0***
[ 3920.148548] Lustre: Mounted lustre-client
[ 3922.540036] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 3939.049569] LustreError: 1962:0:(ldlm_resource.c:982:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8800aa685000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff8800a87bbd00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 3939.490562] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 192.168.204.129@tcp) was lost; in progress operations using this service will fail
[ 3939.491703] LustreError: 11718:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 3939.491706] LustreError: 11718:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages
[ 3939.508024] Lustre: Evicted from MGS (at 192.168.204.129@tcp) after server handle changed from 0x10b27947f6a220c to 0x10b27947f6a27ed
[ 3939.512900] Lustre: Unmounted lustre-client
[ 3942.686207] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 22:59:03 (1746068343)
[ 3959.164217] LustreError: 1962:0:(client.c:3262:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff8800a8036d80 x1830881326826112/t94489280516(94489280516) o101->lustre-MDT0000-mdc-ffff880000025000@192.168.204.129@tcp:12/10 lens 648/608 e 0 to 0 dl 1746068376 ref 2 fl Interpret:RPQU/204/0 rc 301/301 job:'multiop.0' uid:0 gid:0
[ 3962.454984] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3963.143845] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3966.958133] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 22:59:28 (1746068368)
[ 3967.503886] LustreError: lustre-OST0000-osc-ffff880000025000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 3967.509927] Lustre: 1965:0:(llite_lib.c:4118:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.129@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 3967.525690] LustreError: 13882:0:(ldlm_resource.c:982:ldlm_resource_complain()) lustre-OST0000-osc-ffff880000025000: namespace resource [0x240000400:0x85a2:0x0].0x0 (ffff8800a87bb200) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 3967.533835] LustreError: 13882:0:(ldlm_resource.c:982:ldlm_resource_complain()) Skipped 1 previous similar message
[ 3970.839507] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 22:59:31 (1746068371)
[ 3971.360430] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 3971.886054] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 22:59:33 (1746068373)
[ 3972.369594] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 3973.002239] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 22:59:34 (1746068374)
[ 3973.467193] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 3974.009135] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 22:59:35 (1746068375)
[ 3974.501714] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 3975.018763] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 22:59:36 (1746068376)
[ 3975.503526] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 3976.199910] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 22:59:37 (1746068377)
[ 3976.721648] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 3977.363014] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 22:59:38 (1746068378)
[ 3977.869962] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 3978.608910] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 22:59:39 (1746068379)
[ 3979.304539] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 3980.133787] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 22:59:41 (1746068381)
[ 3980.840329] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 3981.805115] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 22:59:42 (1746068382)
[ 3982.530017] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 3983.573650] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 22:59:44 (1746068384)
[ 3984.418529] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 3985.336318] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 22:59:46 (1746068386)
[ 3986.510737] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 3987.347762] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 22:59:48 (1746068388)
[ 3999.127928] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 23:00:00 (1746068400)
[ 4023.262329] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 23:00:24 (1746068424)
[ 4023.496756] Lustre: *** cfs_fail_loc=51b, val=3***
[ 4029.763685] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 23:00:30 (1746068430)
[ 4029.895230] Lustre: *** cfs_fail_loc=51b, val=4***
[ 4036.606780] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 23:00:37 (1746068437)
[ 4036.784974] Lustre: *** cfs_fail_loc=50f, val=3***
[ 4040.864565] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 23:00:41 (1746068441)
[ 4041.042505] Lustre: *** cfs_fail_loc=50f, val=4***
[ 4044.480194] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 23:00:45 (1746068445)
[ 4044.586741] Lustre: *** cfs_fail_loc=510, val=3***
[ 4048.352052] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 23:00:49 (1746068449)
[ 4048.525949] Lustre: *** cfs_fail_loc=51b, val=3***
[ 4108.318914] Lustre: lustre-OST0000-osc-ffff880000025000: Connection to lustre-OST0000 (at 192.168.204.129@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4108.324499] Lustre: Skipped 24 previous similar messages
[ 4110.353443] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 23:01:51 (1746068511)
[ 4110.507006] Lustre: *** cfs_fail_loc=51c, val=3***
[ 4172.667655] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 23:02:53 (1746068573)
[ 4172.827559] LustreError: 25090:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 4175.280007] LustreError: lustre-MDT0000-mdc-ffff880000025000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4175.287496] LustreError: 25106:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff880000025000: inode [0x20000a042:0x8:0x0] mdc close failed: rc = -108
[ 4175.295422] LustreError: 25106:0:(ldlm_resource.c:982:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff880000025000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8800b6f15e00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4175.307893] Lustre: lustre-MDT0000-mdc-ffff880000025000: Connection restored to  (at 192.168.204.129@tcp)
[ 4175.310364] Lustre: Skipped 26 previous similar messages
[ 4176.846893] LustreError: 25090:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 4178.905752] LustreError: 25123:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 4181.321615] LustreError: lustre-MDT0000-mdc-ffff880000025000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4181.337059] LustreError: 25139:0:(ldlm_resource.c:982:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff880000025000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8800b625ab00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4181.349944] LustreError: 25139:0:(ldlm_resource.c:982:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4182.934793] LustreError: 25123:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 4189.439055] LustreError: lustre-MDT0000-mdc-ffff880000025000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4189.663153] LustreError: 25186:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 4192.093225] LustreError: lustre-MDT0000-mdc-ffff880000025000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4192.130315] LustreError: 25207:0:(file.c:6136:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4192.138149] LustreError: 25207:0:(file.c:6136:ll_inode_revalidate_fini()) Skipped 20 previous similar messages
[ 4193.682247] LustreError: 25186:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 4193.694077] LustreError: 25186:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff880000025000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 4193.700656] LustreError: 25186:0:(file.c:247:ll_close_inode_openhandle()) Skipped 4 previous similar messages
[ 4198.592830] LustreError: 25266:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 4201.098007] LustreError: lustre-MDT0000-mdc-ffff880000025000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4201.110701] LustreError: 25282:0:(ldlm_resource.c:982:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff880000025000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff88009ed98b00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4201.144444] LustreError: 25282:0:(ldlm_resource.c:982:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4202.601819] LustreError: 25266:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 4209.116518] LustreError: lustre-MDT0000-mdc-ffff880000025000: operation ldlm_enqueue to node 192.168.204.129@tcp failed: rc = -107
[ 4209.130411] LustreError: Skipped 11 previous similar messages
[ 4209.140384] LustreError: lustre-MDT0000-mdc-ffff880000025000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4209.395714] LustreError: 25329:0:(ldlm_flock.c:856:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 4211.762066] LustreError: lustre-MDT0000-mdc-ffff880000025000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4213.409917] LustreError: 25329:0:(ldlm_flock.c:856:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 4215.861348] LustreError: lustre-MDT0000-mdc-ffff880000025000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4215.903429] LustreError: 25381:0:(file.c:6136:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4215.907899] LustreError: 25381:0:(file.c:6136:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 4217.499052] LustreError: 25360:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff880000025000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 4224.385018] LustreError: 25436:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 4224.391934] LustreError: 25436:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 4226.833412] LustreError: lustre-MDT0000-mdc-ffff880000025000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4228.402807] LustreError: 25436:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 4228.410764] LustreError: 25436:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 4228.417640] LustreError: 25436:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff880000025000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 4234.619478] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 23:03:55 (1746068635)
[ 4251.925803] LustreError: 9052:0:(ldlm_lockd.c:2901:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 4251.930407] LustreError: 9052:0:(ldlm_lockd.c:2901:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 4255.945793] LustreError: 9052:0:(ldlm_lockd.c:2901:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 4255.949653] LustreError: 9052:0:(ldlm_lockd.c:2901:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 4258.999946] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 23:04:19 (1746068659)
[ 4304.408275] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 23:05:05 (1746068705)
[ 4360.494891] Lustre: 27838:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746068707/real 1746068707]  req@ffff8800a8036d80 x1830881326949760/t0(0) o101->lustre-MDT0000-mdc-ffff880000025000@192.168.204.129@tcp:12/10 lens 576/1024 e 0 to 1 dl 1746068762 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'stat.0' uid:0 gid:0
[ 4360.511811] Lustre: 27838:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 13 previous similar messages
[ 4363.756481] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 23:06:04 (1746068764)
[ 4392.065442] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 23:06:32 (1746068792)
[ 4392.427225] Lustre: Mounted lustre-client
[ 4513.653973] LustreError: 29615:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 4513.656875] LustreError: 29615:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 4513.675742] Lustre: Unmounted lustre-client
[ 4515.693467] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 23:08:36 (1746068916)
[ 4515.863746] LustreError: 30289:0:(osc_request.c:2963:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 4000ms
[ 4517.638590] LustreError: lustre-OST0000-osc-ffff880000025000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4517.647612] LustreError: 30373:0:(ldlm_resource.c:982:ldlm_resource_complain()) lustre-OST0000-osc-ffff880000025000: namespace resource [0x240000400:0x85cd:0x0].0x0 (ffff88012c4b0a00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4517.657273] LustreError: 30373:0:(ldlm_resource.c:982:ldlm_resource_complain()) Skipped 4 previous similar messages
[ 4519.871929] LustreError: 30289:0:(osc_request.c:2963:osc_build_rpc()) cfs_fail_timeout id 414 awake
[ 4519.876389] Lustre: 1964:0:(llite_lib.c:4118:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.129@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 4522.136409] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 23:08:43 (1746068923)
[ 4581.140203] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 23:09:42 (1746068982)
[ 4581.669358] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 4582.252177] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 23:09:43 (1746068983)
[ 4639.998839] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 4640.664548] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 23:10:41 (1746069041)
[ 4697.222543] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 23:11:38 (1746069098)
[ 4698.008797] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 4698.759260] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 23:11:39 (1746069099)
[ 4699.543391] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 4700.332532] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 23:11:41 (1746069101)
[ 4714.809968] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 23:11:55 (1746069115)
[ 4719.869571] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4741.021656] Lustre: lustre-MDT0000-mdc-ffff880000025000: Connection to lustre-MDT0000 (at 192.168.204.129@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4741.029185] Lustre: Skipped 17 previous similar messages
[ 4741.501148] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 192.168.204.129@tcp) was lost; in progress operations using this service will fail
[ 4741.506696] LustreError: Skipped 2 previous similar messages
[ 4741.517897] Lustre: Evicted from MGS (at 192.168.204.129@tcp) after server handle changed from 0x10b27947f6a2ff7 to 0x10b27947f6a4b09
[ 4741.521264] Lustre: Skipped 2 previous similar messages
[ 4742.785426] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4743.676448] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4749.729777] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 23:12:30 (1746069150)
[ 4750.829138] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 4751.693315] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 23:12:32 (1746069152)
[ 4763.953152] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 23:12:44 (1746069164)
[ 4776.578949] Lustre: MGC192.168.204.129@tcp: Connection restored to  (at 192.168.204.129@tcp)
[ 4776.582846] Lustre: Skipped 18 previous similar messages
[ 4780.382411] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 23:13:01 (1746069181)
[ 4805.377456] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4806.934479] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4890.853509] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4891.903406] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4914.900412] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4915.700074] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4951.316523] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 23:15:51 (1746069351)
[ 4955.928110] LustreError: lustre-OST0000-osc-ffff880000025000: operation ost_setattr to node 192.168.204.129@tcp failed: rc = -19
[ 4955.944561] LustreError: Skipped 185 previous similar messages
[ 4962.397626] Lustre: 1965:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746069308/real 1746069308]  req@ffff8800a1407800 x1830881333523968/t0(0) o400->lustre-MDT0000-mdc-ffff880000025000@192.168.204.129@tcp:12/10 lens 224/224 e 0 to 1 dl 1746069363 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0
[ 4962.409699] Lustre: 1965:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 14 previous similar messages
[ 4981.951205] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4983.239091] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5134.748055] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 23:18:55 (1746069535)
[ 5251.937051] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 23:20:52 (1746069652)
[ 5252.562428] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 5253.361588] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 23:20:54 (1746069654)
[ 5254.564147] LustreError: lustre-MDT0000-mdc-ffff880000025000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5258.107295] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 23:20:59 (1746069659)
[ 5424.218893] Lustre: Evicted from lustre-OST0000_UUID (at 192.168.204.129@tcp) after server handle changed from 0x10b27947f7a0a14 to 0x10b27947f7a0b17
[ 5424.225812] Lustre: Skipped 5 previous similar messages
[ 5424.229501] LustreError: lustre-OST0000-osc-ffff880000025000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5425.458192] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 23:23:46 (1746069826)
[ 5449.244948] Lustre: lustre-OST0000-osc-ffff880000025000: Connection to lustre-OST0000 (at 192.168.204.129@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5449.260802] Lustre: Skipped 9 previous similar messages
[ 5449.274230] Lustre: lustre-OST0000-osc-ffff880000025000: Connection restored to 192.168.204.129@tcp (at 192.168.204.129@tcp)
[ 5449.278358] Lustre: Skipped 8 previous similar messages
[ 5463.004215] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 23:24:23 (1746069863)
[ 5463.917442] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 5465.232159] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 23:24:25 (1746069865)
[ 5466.246359] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 5467.422604] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 23:24:28 (1746069868)
[ 5468.876058] Lustre: DEBUG MARKER: SKIP: recovery-small test_152 MDS Linux kernel does not support killable semaphore
[ 5469.999551] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 23:24:30 (1746069870)
[ 5499.369780] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 192.168.204.129@tcp) was lost; in progress operations using this service will fail
[ 5499.378147] LustreError: Skipped 5 previous similar messages
[ 5507.019055] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 23:25:07 (1746069907)
[ 5507.804629] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 5508.665096] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 23:25:09 (1746069909)
[ 5509.430818] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 5510.417972] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 23:25:11 (1746069911)
[ 5514.240417] LustreError: 18304:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup
[ 5514.247784] LustreError: 18304:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 5514.289833] Lustre: Unmounted lustre-client
[ 5514.967052] Lustre: Mounted lustre-client
[ 5516.591128] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5535.145014] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: total 2262