[ 1394.077650] LustreError: lustre-OST0000-osc-ffff8800b6d29800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1394.082445] Lustre: 1961:0:(llite_lib.c:4124:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.153@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1395.729390] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 01:33:00 (1732429980)
[ 1396.223536] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1396.592456] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 01:33:01 (1732429981)
[ 1397.010178] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1397.505498] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 01:33:02 (1732429982)
[ 1415.179442] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 192.168.202.153@tcp) was lost; in progress operations using this service will fail
[ 1415.202181] Lustre: Evicted from MGS (at 192.168.202.153@tcp) after server handle changed from 0x5b839e3f0f0ddfe5 to 0x5b839e3f0f0e22f8
[ 1504.653804] LustreError: lustre-MDT0000-mdc-ffff8800b6d29800: operation mds_close to node 192.168.202.153@tcp failed: rc = -19
[ 1504.657693] LustreError: Skipped 4 previous similar messages
[ 1520.402819] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 192.168.202.153@tcp) was lost; in progress operations using this service will fail
[ 1520.407902] Lustre: Evicted from MGS (at 192.168.202.153@tcp) after server handle changed from 0x5b839e3f0f0e22f8 to 0x5b839e3f0f1b83fe
[ 1526.775375] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 01:35:11 (1732430111)
[ 1526.858312] Lustre: *** cfs_fail_loc=305, val=0***
[ 1526.859622] Lustre: Skipped 4 previous similar messages
[ 1559.792014] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1560.162389] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1560.467388] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 192.168.202.153@tcp) was lost; in progress operations using this service will fail
[ 1560.471936] Lustre: Evicted from MGS (at 192.168.202.153@tcp) after server handle changed from 0x5b839e3f0f1b83fe to 0x5b839e3f0f1b89df
[ 1562.717107] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 01:35:47 (1732430147)
[ 1565.475006] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 192.168.202.153@tcp) was lost; in progress operations using this service will fail
[ 1565.479333] Lustre: Evicted from MGS (at 192.168.202.153@tcp) after server handle changed from 0x5b839e3f0f1b89df to 0x5b839e3f0f1b8cef
[ 1565.484163] LustreError: lustre-MDT0000-mdc-ffff8800b6d29800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 1572.298583] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 01:35:56 (1732430156)
[ 1575.492184] LustreError: lustre-OST0000-osc-ffff8800b6d29800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1581.589771] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 01:36:06 (1732430166)
[ 1592.109593] LustreError: lustre-MDT0000-mdc-ffff8800b6d29800: operation mds_close to node 192.168.202.153@tcp failed: rc = -19
[ 1592.112140] LustreError: Skipped 1 previous similar message
[ 1605.539001] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 192.168.202.153@tcp) was lost; in progress operations using this service will fail
[ 1605.544971] Lustre: Evicted from MGS (at 192.168.202.153@tcp) after server handle changed from 0x5b839e3f0f1b8cef to 0x5b839e3f0f1d71a8
[ 1611.341632] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1611.746925] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1688.699370] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 192.168.202.153@tcp) was lost; in progress operations using this service will fail
[ 1688.744461] Lustre: Evicted from MGS (at 192.168.202.153@tcp) after server handle changed from 0x5b839e3f0f1d71a8 to 0x5b839e3f0f28de35
[ 1689.545608] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1689.932226] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1751.705058] LustreError: lustre-MDT0000-mdc-ffff8800b6d29800: operation ldlm_enqueue to node 192.168.202.153@tcp failed: rc = -19
[ 1751.709097] LustreError: Skipped 1 previous similar message
[ 1763.874940] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 192.168.202.153@tcp) was lost; in progress operations using this service will fail
[ 1763.879574] Lustre: Evicted from MGS (at 192.168.202.153@tcp) after server handle changed from 0x5b839e3f0f28de35 to 0x5b839e3f0f336f8b
[ 1769.742770] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1770.163842] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1792.740372] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 01:39:37 (1732430377)
[ 1807.914817] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 1823.976489] Lustre: 15936:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.153@tcp: IR log lustre-cliir failed, not fatal: rc = -5
[ 1824.842695] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 1843.981261] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 192.168.202.153@tcp) was lost; in progress operations using this service will fail
[ 1843.986318] LustreError: Skipped 2 previous similar messages
[ 1843.999327] Lustre: Evicted from MGS (at 192.168.202.153@tcp) after server handle changed from 0x5b839e3f0f379394 to 0x5b839e3f0f3794c8
[ 1844.002541] Lustre: Skipped 2 previous similar messages
[ 1845.252782] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 1876.026062] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 1891.683589] hrtimer: interrupt took 4952336 ns
[ 1897.710188] Lustre: lustre-MDT0000-mdc-ffff8800b6d29800: Connection to lustre-MDT0000 (at 192.168.202.153@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 1897.723398] Lustre: Skipped 17 previous similar messages
[ 1914.096389] Lustre: MGC192.168.202.153@tcp: Connection restored to 192.168.202.153@tcp (at 192.168.202.153@tcp)
[ 1914.100378] Lustre: Skipped 28 previous similar messages
[ 1916.260042] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 1963.423675] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2014.262382] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 192.168.202.153@tcp) was lost; in progress operations using this service will fail
[ 2014.269139] LustreError: Skipped 3 previous similar messages
[ 2014.292486] Lustre: Evicted from MGS (at 192.168.202.153@tcp) after server handle changed from 0x5b839e3f0f3a9e23 to 0x5b839e3f0f3bad2b
[ 2014.302916] Lustre: Skipped 3 previous similar messages
[ 2037.868362] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 01:43:41 (1732430621)
[ 2049.912645] LustreError: lustre-OST0000-osc-ffff8800b6d29800: operation ldlm_enqueue to node 192.168.202.153@tcp failed: rc = -19
[ 2049.918197] LustreError: Skipped 8 previous similar messages
[ 2075.525186] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2077.287996] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 2407.256560] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2409.073801] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 2711.689062] LustreError: lustre-OST0000-osc-ffff8800b6d29800: operation ldlm_enqueue to node 192.168.202.153@tcp failed: rc = -107
[ 2711.699278] LustreError: Skipped 1 previous similar message
[ 2711.702317] Lustre: lustre-OST0000-osc-ffff8800b6d29800: Connection to lustre-OST0000 (at 192.168.202.153@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2711.709088] Lustre: Skipped 4 previous similar messages
[ 2735.158660] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2736.344892] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3005.776349] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 01:59:49 (1732431589)
[ 3022.662501] Lustre: 22058:0:(client.c:2366:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1732431591/real 1732431591]  req@ffff8800a80c6a00 x1816583133353344/t0(0) o101->lustre-MDT0000-mdc-ffff8800b6d29800@192.168.202.153@tcp:12/10 lens 576/1024 e 0 to 1 dl 1732431607 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0
[ 3022.688862] Lustre: 22058:0:(client.c:2366:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
[ 3022.708988] Lustre: lustre-MDT0000-mdc-ffff8800b6d29800: Connection restored to 192.168.202.153@tcp (at 192.168.202.153@tcp)
[ 3022.713815] Lustre: Skipped 6 previous similar messages
[ 3026.175586] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 02:00:10 (1732431610)
[ 3046.809674] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 02:00:30 (1732431630)
[ 3068.637423] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 02:00:52 (1732431652)
[ 3068.901800] Lustre: Mounted lustre-client
[ 3098.956757] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 192.168.202.153@tcp) was lost; in progress operations using this service will fail
[ 3098.978975] Lustre: Evicted from MGS (at 192.168.202.153@tcp) after server handle changed from 0x5b839e3f0f3bad2b to 0x5b839e3f0f5dbdb8
[ 3099.944286] Lustre: 1962:0:(client.c:2366:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1732431668/real 1732431668]  req@ffff8800a8329f80 x1816583133373568/t0(0) o400->lustre-MDT0000-mdc-ffff8800b6d29800@192.168.202.153@tcp:12/10 lens 224/224 e 0 to 1 dl 1732431684 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0
[ 3099.960375] Lustre: 1962:0:(client.c:2366:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 3101.134598] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3102.132805] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3103.743057] Lustre: Unmounted lustre-client
[ 3107.448271] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 02:01:31 (1732431691)
[ 3259.737470] Lustre: 1962:0:(client.c:2366:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1732431828/real 1732431828]  req@ffff8800a86a6a00 x1816583133433088/t0(0) o4->lustre-OST0000-osc-ffff8800b6d29800@192.168.202.153@tcp:6/4 lens 488/448 e 0 to 1 dl 1732431844 ref 2 fl Rpc:XQr/202/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0
[ 3259.753952] Lustre: 1962:0:(client.c:2366:ptlrpc_expire_one_request()) Skipped 93 previous similar messages
[ 3323.801303] Lustre: lustre-OST0000-osc-ffff8800b6d29800: Connection to lustre-OST0000 (at 192.168.202.153@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3323.810718] Lustre: Skipped 17 previous similar messages
[ 3400.175536] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 02:06:24 (1732431984)
[ 3444.743676] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 02:07:08 (1732432028)
[ 3446.325945] Lustre: Unmounted lustre-client
[ 3458.309392] Lustre: Mounted lustre-client
[ 3462.397385] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 02:07:26 (1732432046)
[ 3462.720338] LustreError: 28673:0:(mdc_locks.c:1240:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 3463.731948] LustreError: 28673:0:(mdc_locks.c:1240:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 3463.838151] Lustre: *** cfs_fail_loc=305, val=0***
[ 3483.699877] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 02:07:47 (1732432067)
[ 3483.908830] Lustre: Mounted lustre-client
[ 3484.484419] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 3494.536668] Lustre: Unmounted lustre-client
[ 3497.876352] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 02:08:02 (1732432082)
[ 3561.553834] LustreError: lustre-MDT0000-mdc-ffff8801302ec000: operation mds_reint to node 192.168.202.153@tcp failed: rc = -19
[ 3579.041878] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 192.168.202.153@tcp) was lost; in progress operations using this service will fail
[ 3579.052007] Lustre: Evicted from MGS (at 192.168.202.153@tcp) after server handle changed from 0x5b839e3f0f5dd29c to 0x5b839e3f0f608dbc
[ 3669.496702] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 02:10:53 (1732432253)
[ 3672.237580] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 3679.214414] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 192.168.202.153@tcp) was lost; in progress operations using this service will fail
[ 3679.231481] Lustre: Evicted from MGS (at 192.168.202.153@tcp) after server handle changed from 0x5b839e3f0f608dbc to 0x5b839e3f0f654d4d
[ 3679.254458] Lustre: MGC192.168.202.153@tcp: Connection restored to 192.168.202.153@tcp (at 192.168.202.153@tcp)
[ 3679.254568] LustreError: lustre-MDT0000-mdc-ffff8801302ec000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 3679.263577] Lustre: Skipped 25 previous similar messages
[ 3687.543381] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 02:11:11 (1732432271)
[ 3687.781824] Lustre: Mounted lustre-client
[ 3694.694967] LustreError: lustre-OST0000-osc-ffff8801302ec000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 3704.739592] Lustre: 1961:0:(client.c:2366:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1732432273/real 1732432273]  req@ffff8800b58e8e00 x1816583138090624/t0(0) o103->lustre-OST0000-osc-ffff880136bcb800@192.168.202.153@tcp:17/18 lens 328/224 e 0 to 1 dl 1732432289 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0
[ 3704.758563] Lustre: 1961:0:(client.c:2366:ptlrpc_expire_one_request()) Skipped 112 previous similar messages
[ 3705.149236] Lustre: Unmounted lustre-client
[ 3709.254423] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 02:11:33 (1732432293)
[ 3715.014342] LustreError: lustre-MDT0000-mdc-ffff8801302ec000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 3715.024177] LustreError: 1069:0:(file.c:5733:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 3718.903407] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 02:11:43 (1732432303)
[ 3719.146519] LustreError: 1814:0:(recover.c:338:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 3722.712211] LustreError: lustre-MDT0000-mdc-ffff8801302ec000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 3722.720540] LustreError: 1836:0:(import.c:307:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 3722.731296] LustreError: 1814:0:(recover.c:338:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=1418
[ 3722.737076] LustreError: 1814:0:(import.c:705:ptlrpc_connect_import_locked()) already connecting
[ 3722.751758] LustreError: 1841:0:(file.c:5733:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3723.822798] LustreError: 1848:0:(file.c:5733:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3723.829921] LustreError: 1848:0:(file.c:5733:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 3726.057098] LustreError: 1872:0:(file.c:5733:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3726.063855] LustreError: 1872:0:(file.c:5733:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 3730.544556] LustreError: 1919:0:(file.c:5733:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3730.567518] LustreError: 1919:0:(file.c:5733:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 3737.260398] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 02:12:01 (1732432321)
[ 3760.874259] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3761.931354] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3768.489129] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 02:12:32 (1732432352)
[ 3792.122292] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3793.327385] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3800.474417] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 02:13:04 (1732432384)
[ 3849.438336] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3850.381673] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3855.819358] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 02:13:59 (1732432439)
[ 3878.855677] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3879.816192] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3883.017665] Lustre: Unmounted lustre-client
[ 3913.865312] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3915.796389] Lustre: Mounted lustre-client
[ 3919.720661] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 02:15:03 (1732432503)
[ 3921.062298] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 3922.172699] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 02:15:06 (1732432506)
[ 3925.800753] Lustre: lustre-OST0000-osc-ffff8800b5927800: Connection to lustre-OST0000 (at 192.168.202.153@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3925.807241] Lustre: Skipped 18 previous similar messages
[ 3937.018138] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 02:15:21 (1732432521)
[ 3937.771817] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 3938.648689] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 02:15:22 (1732432522)
[ 3939.289337] Lustre: *** cfs_fail_loc=805, val=0***
[ 3939.310054] Lustre: Mounted lustre-client
[ 3941.581277] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 3958.287226] LustreError: 1960:0:(ldlm_resource.c:1149:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8801302eb800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff88012ab2d400) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 3958.754708] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 192.168.202.153@tcp) was lost; in progress operations using this service will fail
[ 3958.767901] Lustre: Evicted from MGS (at 192.168.202.153@tcp) after server handle changed from 0x5b839e3f0f6557f7 to 0x5b839e3f0f655de6
[ 3958.781407] Lustre: Unmounted lustre-client
[ 3963.320233] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 02:15:47 (1732432547)
[ 3979.981560] LustreError: 1960:0:(client.c:3312:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff88013688a680 x1816583138159616/t85899345923(85899345923) o101->lustre-MDT0000-mdc-ffff8800b5927800@192.168.202.153@tcp:12/10 lens 664/608 e 0 to 0 dl 1732432580 ref 2 fl Interpret:RPQU/204/0 rc 301/301 job:'multiop.0' uid:0 gid:0
[ 3985.940006] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3986.954827] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3992.507659] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 02:16:16 (1732432576)
[ 3993.277299] LustreError: lustre-OST0000-osc-ffff8800b5927800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 3993.285187] Lustre: 1963:0:(llite_lib.c:4124:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.153@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 3993.301754] LustreError: 13670:0:(ldlm_resource.c:1149:ldlm_resource_complain()) lustre-OST0000-osc-ffff8800b5927800: namespace resource [0x240000400:0x95c2:0x0].0x0 (ffff8800b5bc1f00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 3993.311959] LustreError: 13670:0:(ldlm_resource.c:1149:ldlm_resource_complain()) Skipped 1 previous similar message
[ 3998.117682] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 02:16:22 (1732432582)
[ 3998.960234] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 3999.874199] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 02:16:24 (1732432584)
[ 4000.767778] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 4001.815503] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 02:16:25 (1732432585)
[ 4002.734528] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 4004.105348] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 02:16:27 (1732432587)
[ 4005.093625] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 4006.527172] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 02:16:30 (1732432590)
[ 4007.430984] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 4008.525136] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 02:16:32 (1732432592)
[ 4009.496755] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 4010.535379] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 02:16:34 (1732432594)
[ 4011.514200] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 4012.571224] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 02:16:36 (1732432596)
[ 4013.442685] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 4014.972048] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 02:16:38 (1732432598)
[ 4016.035432] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 4017.287658] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 02:16:41 (1732432601)
[ 4018.259592] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 4019.333166] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 02:16:43 (1732432603)
[ 4020.272935] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 4021.299132] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 02:16:45 (1732432605)
[ 4022.711489] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 4023.852009] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 02:16:48 (1732432608)
[ 4040.910638] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 02:17:04 (1732432624)
[ 4065.997101] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 02:17:30 (1732432650)
[ 4066.441604] Lustre: *** cfs_fail_loc=51b, val=3***
[ 4074.516113] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 02:17:38 (1732432658)
[ 4074.770257] Lustre: *** cfs_fail_loc=51b, val=4***
[ 4082.483211] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 02:17:46 (1732432666)
[ 4082.653840] Lustre: *** cfs_fail_loc=50f, val=3***
[ 4088.518123] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 02:17:52 (1732432672)
[ 4088.696321] Lustre: *** cfs_fail_loc=50f, val=4***
[ 4094.266419] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 02:17:58 (1732432678)
[ 4094.424634] Lustre: *** cfs_fail_loc=510, val=3***
[ 4099.696713] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 02:18:03 (1732432683)
[ 4100.017687] Lustre: *** cfs_fail_loc=51b, val=3***
[ 4107.504562] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 02:18:11 (1732432691)
[ 4169.438024] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 02:19:13 (1732432753)
[ 4169.653987] LustreError: 24820:0:(ldlm_flock.c:658:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 4172.182552] LustreError: lustre-MDT0000-mdc-ffff8800b5927800: operation ldlm_enqueue to node 192.168.202.153@tcp failed: rc = -107
[ 4172.194162] LustreError: Skipped 6 previous similar messages
[ 4172.207852] LustreError: lustre-MDT0000-mdc-ffff8800b5927800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4172.218964] LustreError: 24839:0:(file.c:262:ll_close_inode_openhandle()) lustre-clilmv-ffff8800b5927800: inode [0x20000a042:0xb:0x0] mdc close failed: rc = -108
[ 4172.227907] LustreError: 24839:0:(ldlm_resource.c:1149:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8800b5927800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8800b5a3b600) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4173.669016] LustreError: 24820:0:(ldlm_flock.c:658:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 4175.729245] LustreError: 24856:0:(ldlm_flock.c:658:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 4178.195406] LustreError: lustre-MDT0000-mdc-ffff8800b5927800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4178.202690] LustreError: 24875:0:(ldlm_resource.c:1149:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8800b5927800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8800b5a3bf00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4178.216085] LustreError: 24875:0:(ldlm_resource.c:1149:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4178.228121] LustreError: 24880:0:(file.c:5733:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4178.233142] LustreError: 24880:0:(file.c:5733:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4179.739036] LustreError: 24856:0:(ldlm_flock.c:658:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 4183.821399] LustreError: 24893:0:(ldlm_flock.c:658:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 4186.313607] LustreError: lustre-MDT0000-mdc-ffff8800b5927800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4186.322485] LustreError: 24912:0:(ldlm_resource.c:1149:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8800b5927800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8800b5a3ba00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4186.331585] LustreError: 24912:0:(ldlm_resource.c:1149:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4187.840036] LustreError: 24893:0:(ldlm_flock.c:658:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 4187.902402] LustreError: 24926:0:(ldlm_flock.c:663:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 4190.362499] LustreError: lustre-MDT0000-mdc-ffff8800b5927800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4191.923302] LustreError: 24926:0:(ldlm_flock.c:663:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 4191.928451] LustreError: 24926:0:(file.c:262:ll_close_inode_openhandle()) lustre-clilmv-ffff8800b5927800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 4191.934252] LustreError: 24926:0:(file.c:262:ll_close_inode_openhandle()) Skipped 3 previous similar messages
[ 4197.063487] LustreError: 25011:0:(ldlm_flock.c:663:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 4199.505696] LustreError: lustre-MDT0000-mdc-ffff8800b5927800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4199.521418] LustreError: 25029:0:(ldlm_resource.c:1149:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8800b5927800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff8800b6f93c00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4199.529504] LustreError: 25029:0:(ldlm_resource.c:1149:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4201.078113] LustreError: 25011:0:(ldlm_flock.c:663:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 4205.127387] LustreError: 25047:0:(ldlm_flock.c:663:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 4207.699497] LustreError: lustre-MDT0000-mdc-ffff8800b5927800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4209.150051] LustreError: 25047:0:(ldlm_flock.c:663:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 4211.760846] LustreError: lustre-MDT0000-mdc-ffff8800b5927800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4215.855561] LustreError: lustre-MDT0000-mdc-ffff8800b5927800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4215.891365] LustreError: 25138:0:(file.c:5733:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4215.897139] LustreError: 25138:0:(file.c:5733:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4217.343235] LustreError: 25114:0:(file.c:262:ll_close_inode_openhandle()) lustre-clilmv-ffff8800b5927800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 4224.593374] LustreError: 25192:0:(ldlm_flock.c:658:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 4224.597665] LustreError: 25192:0:(ldlm_flock.c:658:ldlm_flock_completion_ast()) Skipped 2 previous similar messages
[ 4227.076673] LustreError: lustre-MDT0000-mdc-ffff8800b5927800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4228.608033] LustreError: 25192:0:(ldlm_flock.c:658:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 4228.610909] LustreError: 25192:0:(ldlm_flock.c:658:ldlm_flock_completion_ast()) Skipped 2 previous similar messages
[ 4228.613735] LustreError: 25192:0:(file.c:262:ll_close_inode_openhandle()) lustre-clilmv-ffff8800b5927800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 4235.085078] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 02:20:18 (1732432818)
[ 4261.282619] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 02:20:45 (1732432845)
[ 4304.545849] Lustre: lustre-MDT0000-mdc-ffff8800b5927800: Connection restored to 192.168.202.153@tcp (at 192.168.202.153@tcp)
[ 4304.552779] Lustre: Skipped 27 previous similar messages
[ 4308.439354] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 02:21:32 (1732432892)
[ 4365.561309] Lustre: 27599:0:(client.c:2366:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1732432894/real 1732432894]  req@ffff8800a7937480 x1816583138283776/t0(0) o101->lustre-MDT0000-mdc-ffff8800b5927800@192.168.202.153@tcp:12/10 lens 576/1024 e 0 to 1 dl 1732432950 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'stat.0' uid:0 gid:0
[ 4365.575026] Lustre: 27599:0:(client.c:2366:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
[ 4369.552193] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 02:22:33 (1732432953)
[ 4398.224240] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 02:23:02 (1732432982)
[ 4398.527777] Lustre: Mounted lustre-client
[ 4520.109471] Lustre: Unmounted lustre-client
[ 4524.203304] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 02:25:08 (1732433108)
[ 4524.498569] LustreError: 30093:0:(osc_request.c:2930:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 4000ms
[ 4524.502256] LustreError: 30093:0:(osc_request.c:2930:osc_build_rpc()) Skipped 2 previous similar messages
[ 4526.686441] Lustre: lustre-OST0000-osc-ffff8800b5927800: Connection to lustre-OST0000 (at 192.168.202.153@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4526.695012] Lustre: Skipped 18 previous similar messages
[ 4526.703927] LustreError: lustre-OST0000-osc-ffff8800b5927800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4526.713220] LustreError: 30175:0:(ldlm_resource.c:1149:ldlm_resource_complain()) lustre-OST0000-osc-ffff8800b5927800: namespace resource [0x240000400:0x95ed:0x0].0x0 (ffff8800a8620f00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4526.721977] LustreError: 30175:0:(ldlm_resource.c:1149:ldlm_resource_complain()) Skipped 6 previous similar messages
[ 4528.509825] LustreError: 30093:0:(osc_request.c:2930:osc_build_rpc()) cfs_fail_timeout id 414 awake
[ 4528.522571] LustreError: 30093:0:(osc_request.c:2930:osc_build_rpc()) Skipped 2 previous similar messages
[ 4528.527070] Lustre: 1963:0:(llite_lib.c:4124:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.153@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 4532.205270] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 02:25:16 (1732433116)
[ 4593.306765] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 02:26:17 (1732433177)
[ 4593.892601] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 4594.606052] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 02:26:18 (1732433178)
[ 4654.715582] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 4655.733678] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 02:27:19 (1732433239)
[ 4713.676655] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 02:28:17 (1732433297)
[ 4714.643606] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 4715.796017] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 02:28:19 (1732433299)
[ 4716.822081] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 4718.042651] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 02:28:21 (1732433301)
[ 4732.507442] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 02:28:36 (1732433316)
[ 4737.441234] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4757.545308] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 192.168.202.153@tcp) was lost; in progress operations using this service will fail
[ 4757.554879] LustreError: Skipped 2 previous similar messages
[ 4757.566660] Lustre: Evicted from MGS (at 192.168.202.153@tcp) after server handle changed from 0x5b839e3f0f6565e9 to 0x5b839e3f0f65803e
[ 4757.570751] Lustre: Skipped 2 previous similar messages
[ 4760.224533] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4761.144960] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4766.740427] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 02:29:11 (1732433351)
[ 4768.153179] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 4768.908230] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 02:29:13 (1732433353)
[ 4779.470730] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 02:29:23 (1732433363)
[ 4800.798725] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 02:29:45 (1732433385)
[ 4803.495119] LustreError: lustre-OST0000-osc-ffff8800b5927800: operation ost_setattr to node 192.168.202.153@tcp failed: rc = -19
[ 4803.498611] LustreError: Skipped 9 previous similar messages
[ 4822.440080] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4823.517587] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4903.616538] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4904.139983] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4922.691581] Lustre: lustre-MDT0000-mdc-ffff8800b5927800: Connection restored to 192.168.202.153@tcp (at 192.168.202.153@tcp)
[ 4922.694676] Lustre: Skipped 12 previous similar messages
[ 4924.055533] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4924.633486] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4947.425310] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 02:32:11 (1732433531)
[ 4967.898021] Lustre: 1963:0:(client.c:2366:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1732433497/real 1732433497]  req@ffff88009b707800 x1816583144897408/t0(0) o400->lustre-MDT0000-mdc-ffff8800b5927800@192.168.202.153@tcp:12/10 lens 224/224 e 0 to 1 dl 1732433552 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0
[ 4967.934778] Lustre: 1963:0:(client.c:2366:ptlrpc_expire_one_request()) Skipped 13 previous similar messages
[ 4971.145295] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4971.795856] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5097.274144] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 02:34:41 (1732433681)
[ 5158.213664] Lustre: lustre-MDT0000-mdc-ffff8800b5927800: Connection to lustre-MDT0000 (at 192.168.202.153@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5158.223644] Lustre: Skipped 10 previous similar messages
[ 5179.843543] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 02:36:04 (1732433764)
[ 5180.277220] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 5180.809969] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 02:36:05 (1732433765)
[ 5181.407916] LustreError: lustre-MDT0000-mdc-ffff8800b5927800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5183.626960] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 02:36:08 (1732433768)
[ 5349.766054] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 02:38:54 (1732433934)
[ 5382.115498] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 02:39:26 (1732433966)
[ 5382.480565] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 5382.882275] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 02:39:27 (1732433967)
[ 5383.250294] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 5383.625583] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 02:39:28 (1732433968)
[ 5384.220408] Lustre: DEBUG MARKER: SKIP: recovery-small test_152 MDS Linux kernel does not support killable semaphore
[ 5384.629553] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 02:39:29 (1732433969)
[ 5414.418866] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 192.168.202.153@tcp) was lost; in progress operations using this service will fail
[ 5414.423320] LustreError: Skipped 5 previous similar messages
[ 5414.427363] Lustre: Evicted from MGS (at 192.168.202.153@tcp) after server handle changed from 0x5b839e3f0f7029e3 to 0x5b839e3f0f703600
[ 5414.430226] Lustre: Skipped 5 previous similar messages
[ 5419.899302] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 02:40:04 (1732434004)
[ 5420.311523] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 5420.747166] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 02:40:05 (1732434005)
[ 5421.139287] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 5421.578952] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 02:40:06 (1732434006)
[ 5423.772957] Lustre: Unmounted lustre-client
[ 5424.127284] Lustre: Mounted lustre-client
[ 5424.998128] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5439.139634] LustreError: lustre-MDT0000-mdc-ffff8800b5995000: operation ldlm_enqueue to node 192.168.202.153@tcp failed: rc = -107
[ 5439.144298] LustreError: Skipped 588 previous similar messages
[ 5439.703201] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: total 2252