[ 1465.108177] LustreError: lustre-OST0000-osc-ffff8800a86a9000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1465.132608] Lustre: 1971:0:(llite_lib.c:4117:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.129@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1468.320468] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 18:36:56 (1746139016)
[ 1469.297971] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1470.033630] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 18:36:58 (1746139018)
[ 1470.849572] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1471.750644] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 18:36:59 (1746139019)
[ 1490.660507] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 192.168.204.129@tcp) was lost; in progress operations using this service will fail
[ 1490.674885] Lustre: Evicted from MGS (at 192.168.204.129@tcp) after server handle changed from 0x2fbf3af6764ce212 to 0x2fbf3af6764d0574
[ 1582.287925] LustreError: lustre-MDT0000-mdc-ffff8800a86a9000: operation mds_reint to node 192.168.204.129@tcp failed: rc = -19
[ 1582.292240] LustreError: Skipped 3 previous similar messages
[ 1595.882881] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 192.168.204.129@tcp) was lost; in progress operations using this service will fail
[ 1595.942148] Lustre: Evicted from MGS (at 192.168.204.129@tcp) after server handle changed from 0x2fbf3af6764d0574 to 0x2fbf3af6765204d0
[ 1604.184989] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 18:39:12 (1746139152)
[ 1604.545545] Lustre: *** cfs_fail_loc=305, val=0***
[ 1604.547213] Lustre: Skipped 4 previous similar messages
[ 1641.033772] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 192.168.204.129@tcp) was lost; in progress operations using this service will fail
[ 1641.050779] Lustre: Evicted from MGS (at 192.168.204.129@tcp) after server handle changed from 0x2fbf3af6765204d0 to 0x2fbf3af676520b2f
[ 1643.996121] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1644.787550] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1648.804592] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 18:39:56 (1746139196)
[ 1656.058860] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 192.168.204.129@tcp) was lost; in progress operations using this service will fail
[ 1656.083543] Lustre: Evicted from MGS (at 192.168.204.129@tcp) after server handle changed from 0x2fbf3af676520b2f to 0x2fbf3af676520fa4
[ 1656.111376] LustreError: lustre-MDT0000-mdc-ffff8800a86a9000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 1665.736181] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 18:40:13 (1746139213)
[ 1676.084454] LustreError: lustre-OST0000-osc-ffff8800a86a9000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1681.711663] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 18:40:29 (1746139229)
[ 1693.157601] LustreError: lustre-MDT0000-mdc-ffff8800a86a9000: operation mds_close to node 192.168.204.129@tcp failed: rc = -19
[ 1693.169412] LustreError: Skipped 2 previous similar messages
[ 1711.160467] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 192.168.204.129@tcp) was lost; in progress operations using this service will fail
[ 1711.198894] Lustre: Evicted from MGS (at 192.168.204.129@tcp) after server handle changed from 0x2fbf3af676520fa4 to 0x2fbf3af67652a704
[ 1713.335045] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1714.479421] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1777.201739] Lustre: lustre-MDT0000-mdc-ffff8800a86a9000: Connection to lustre-MDT0000 (at 192.168.204.129@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 1777.212618] Lustre: Skipped 16 previous similar messages
[ 1791.265802] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 192.168.204.129@tcp) was lost; in progress operations using this service will fail
[ 1791.282433] Lustre: Evicted from MGS (at 192.168.204.129@tcp) after server handle changed from 0x2fbf3af67652a704 to 0x2fbf3af67656145e
[ 1791.289441] Lustre: MGC192.168.204.129@tcp: Connection restored to 192.168.204.129@tcp (at 192.168.204.129@tcp)
[ 1791.292593] Lustre: Skipped 21 previous similar messages
[ 1798.242475] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1799.100460] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1861.991072] LustreError: lustre-MDT0000-mdc-ffff8800a86a9000: operation ldlm_enqueue to node 192.168.204.129@tcp failed: rc = -19
[ 1861.995029] LustreError: Skipped 1 previous similar message
[ 1876.497195] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 192.168.204.129@tcp) was lost; in progress operations using this service will fail
[ 1876.504779] Lustre: Evicted from MGS (at 192.168.204.129@tcp) after server handle changed from 0x2fbf3af67656145e to 0x2fbf3af676593b87
[ 1883.507944] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1884.626573] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1909.511737] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 18:44:17 (1746139457)
[ 1930.361650] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 1946.677152] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 192.168.204.129@tcp) was lost; in progress operations using this service will fail
[ 1946.692807] LustreError: Skipped 1 previous similar message
[ 1946.710950] Lustre: Evicted from MGS (at 192.168.204.129@tcp) after server handle changed from 0x2fbf3af6765a5a3a to 0x2fbf3af6765a63a2
[ 1946.720631] Lustre: Skipped 1 previous similar message
[ 1951.652402] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 1975.207901] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2003.863370] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2041.449955] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2081.921399] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 192.168.204.129@tcp) was lost; in progress operations using this service will fail
[ 2081.927543] LustreError: Skipped 3 previous similar messages
[ 2081.938819] Lustre: Evicted from MGS (at 192.168.204.129@tcp) after server handle changed from 0x2fbf3af6765c84d0 to 0x2fbf3af6765dc14a
[ 2081.943605] Lustre: Skipped 3 previous similar messages
[ 2084.210893] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2155.216886] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 18:48:23 (1746139703)
[ 2166.732980] LustreError: lustre-OST0000-osc-ffff8800a86a9000: operation ost_write to node 192.168.204.129@tcp failed: rc = -19
[ 2166.743765] LustreError: Skipped 10 previous similar messages
[ 2189.159984] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2190.430839] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 2498.121674] Lustre: lustre-OST0000-osc-ffff8800a86a9000: Connection to lustre-OST0000 (at 192.168.204.129@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2498.130495] Lustre: Skipped 9 previous similar messages
[ 2519.951669] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2521.023786] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 2828.902992] LustreError: lustre-OST0000-osc-ffff8800a86a9000: operation ldlm_enqueue to node 192.168.204.129@tcp failed: rc = -19
[ 2828.908313] LustreError: Skipped 3 previous similar messages
[ 2852.605502] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2853.654054] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3122.610994] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 19:04:30 (1746140670)
[ 3139.478450] Lustre: 22368:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746140672/real 1746140672]  req@ffff8800a98ad500 x1830958018020992/t0(0) o101->lustre-MDT0000-mdc-ffff8800a86a9000@192.168.204.129@tcp:12/10 lens 576/1024 e 0 to 1 dl 1746140688 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0
[ 3139.494967] Lustre: 22368:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
[ 3139.500854] Lustre: lustre-MDT0000-mdc-ffff8800a86a9000: Connection to lustre-MDT0000 (at 192.168.204.129@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3139.507801] Lustre: Skipped 1 previous similar message
[ 3139.522545] Lustre: lustre-MDT0000-mdc-ffff8800a86a9000: Connection restored to 192.168.204.129@tcp (at 192.168.204.129@tcp)
[ 3139.526890] Lustre: Skipped 18 previous similar messages
[ 3142.563137] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 19:04:50 (1746140690)
[ 3163.591912] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 19:05:11 (1746140711)
[ 3184.162111] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 19:05:32 (1746140732)
[ 3184.410973] Lustre: Mounted lustre-client
[ 3209.441769] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 192.168.204.129@tcp) was lost; in progress operations using this service will fail
[ 3209.448679] LustreError: Skipped 1 previous similar message
[ 3209.463104] Lustre: Evicted from MGS (at 192.168.204.129@tcp) after server handle changed from 0x2fbf3af6765f4013 to 0x2fbf3af6768a3a0d
[ 3209.469927] Lustre: Skipped 1 previous similar message
[ 3214.711912] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3215.397140] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3215.445349] Lustre: 1971:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746140748/real 1746140748]  req@ffff88012b4db480 x1830958018037248/t0(0) o400->lustre-MDT0000-mdc-ffff8800a86a9000@192.168.204.129@tcp:12/10 lens 224/224 e 0 to 1 dl 1746140764 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0
[ 3215.461639] Lustre: 1971:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 3216.862856] LustreError: 25369:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 3216.865653] LustreError: 25369:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 3216.898397] Lustre: Unmounted lustre-client
[ 3219.580189] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 19:06:07 (1746140767)
[ 3258.502453] Lustre: lustre-OST0000-osc-ffff8800a86a9000: Connection restored to  (at 192.168.204.129@tcp)
[ 3258.506975] Lustre: Skipped 5 previous similar messages
[ 3370.563541] Lustre: 1971:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746140903/real 1746140903]  req@ffff88012b4d9c00 x1830958018096128/t0(0) o4->lustre-OST0000-osc-ffff8800a86a9000@192.168.204.129@tcp:6/4 lens 488/448 e 0 to 1 dl 1746140919 ref 2 fl Rpc:XQr/202/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0
[ 3370.582979] Lustre: 1971:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 52 previous similar messages
[ 3418.627478] Lustre: lustre-OST0000-osc-ffff8800a86a9000: Connection restored to  (at 192.168.204.129@tcp)
[ 3418.633069] Lustre: Skipped 9 previous similar messages
[ 3509.414158] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 19:10:57 (1746141057)
[ 3554.264496] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 19:11:42 (1746141102)
[ 3555.853352] LustreError: 27921:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup
[ 3555.855882] LustreError: 27921:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 3555.899940] Lustre: Unmounted lustre-client
[ 3567.909591] Lustre: Mounted lustre-client
[ 3571.331989] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 19:11:59 (1746141119)
[ 3571.531645] LustreError: 28977:0:(mdc_locks.c:1333:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 3572.648064] Lustre: *** cfs_fail_loc=305, val=0***
[ 3591.583266] LustreError: 28977:0:(mdc_locks.c:1333:mdc_finish_intent_lock()) cfs_fail_timeout id 801 awake
[ 3594.674039] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 19:12:22 (1746141142)
[ 3594.871059] Lustre: Mounted lustre-client
[ 3595.300840] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 3605.331867] LustreError: 29720:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 3605.336831] LustreError: 29720:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages
[ 3605.353714] Lustre: Unmounted lustre-client
[ 3608.170855] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 19:12:36 (1746141156)
[ 3655.907147] LustreError: lustre-MDT0000-mdc-ffff8800a7c5c000: operation mds_reint to node 192.168.204.129@tcp failed: rc = -19
[ 3669.991600] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 192.168.204.129@tcp) was lost; in progress operations using this service will fail
[ 3670.007976] Lustre: Evicted from MGS (at 192.168.204.129@tcp) after server handle changed from 0x2fbf3af6768a4693 to 0x2fbf3af6768d0587
[ 3749.344875] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 19:14:57 (1746141297)
[ 3751.947544] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 3760.161695] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 192.168.204.129@tcp) was lost; in progress operations using this service will fail
[ 3760.163718] Lustre: lustre-MDT0000-mdc-ffff8800a7c5c000: Connection to lustre-MDT0000 (at 192.168.204.129@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3760.163720] Lustre: Skipped 24 previous similar messages
[ 3760.181707] Lustre: Evicted from MGS (at 192.168.204.129@tcp) after server handle changed from 0x2fbf3af6768d0587 to 0x2fbf3af67691ba9f
[ 3760.185299] Lustre: MGC192.168.204.129@tcp: Connection restored to 192.168.204.129@tcp (at 192.168.204.129@tcp)
[ 3760.188861] Lustre: Skipped 9 previous similar messages
[ 3760.192036] LustreError: lustre-MDT0000-mdc-ffff8800a7c5c000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 3766.128933] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 19:15:14 (1746141314)
[ 3766.295007] Lustre: Mounted lustre-client
[ 3771.306901] LustreError: lustre-OST0000-osc-ffff8800a7c5c000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 3782.309809] Lustre: 1970:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746141315/real 1746141315]  req@ffff8800a7db5500 x1830958022742784/t0(0) o103->lustre-OST0000-osc-ffff88012a446800@192.168.204.129@tcp:17/18 lens 328/224 e 0 to 1 dl 1746141331 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0
[ 3782.330469] Lustre: 1970:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 90 previous similar messages
[ 3782.613970] LustreError: 596:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 3782.616638] LustreError: 596:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 3782.643037] Lustre: Unmounted lustre-client
[ 3785.736524] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 19:15:33 (1746141333)
[ 3791.059386] LustreError: lustre-MDT0000-mdc-ffff8800a7c5c000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 3791.074235] LustreError: 1329:0:(file.c:6135:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 3794.493150] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 19:15:42 (1746141342)
[ 3794.619477] LustreError: 2112:0:(recover.c:322:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 3798.186545] LustreError: lustre-MDT0000-mdc-ffff8800a7c5c000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 3798.194426] LustreError: 2131:0:(import.c:287:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 3798.198070] LustreError: 2112:0:(recover.c:322:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=1424
[ 3798.203344] LustreError: 2112:0:(import.c:709:ptlrpc_connect_import_locked()) already connecting
[ 3798.211055] LustreError: 2136:0:(file.c:6135:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3799.252108] LustreError: 2143:0:(file.c:6135:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3799.257273] LustreError: 2143:0:(file.c:6135:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 3801.413279] LustreError: 2167:0:(file.c:6135:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3801.424971] LustreError: 2167:0:(file.c:6135:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 3805.702869] LustreError: 2214:0:(file.c:6135:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3805.713253] LustreError: 2214:0:(file.c:6135:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 3811.923464] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 19:16:00 (1746141360)
[ 3833.733824] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3834.406401] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3839.093869] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 19:16:27 (1746141387)
[ 3859.280674] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3859.964840] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3864.904535] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 19:16:53 (1746141413)
[ 3910.839638] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3911.744424] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3915.627631] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 19:17:43 (1746141463)
[ 3935.606511] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3936.232052] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3938.856999] LustreError: 8256:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup
[ 3938.860350] LustreError: 8256:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 3938.878686] Lustre: Unmounted lustre-client
[ 3964.405698] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3966.137641] Lustre: Mounted lustre-client
[ 3968.552748] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 19:18:36 (1746141516)
[ 3969.507472] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 3970.170865] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 19:18:38 (1746141518)
[ 3981.316588] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 19:18:49 (1746141529)
[ 3981.990796] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 3982.792861] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 19:18:50 (1746141530)
[ 3983.385127] Lustre: *** cfs_fail_loc=805, val=0***
[ 3983.404847] Lustre: Mounted lustre-client
[ 3985.558255] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4002.103001] LustreError: 1968:0:(ldlm_resource.c:982:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8800aa565000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff88009d0ef800) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4002.600644] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 192.168.204.129@tcp) was lost; in progress operations using this service will fail
[ 4002.601981] LustreError: 11777:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 4002.601983] LustreError: 11777:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages
[ 4002.623737] Lustre: Evicted from MGS (at 192.168.204.129@tcp) after server handle changed from 0x2fbf3af67691c49a to 0x2fbf3af67691ca7b
[ 4002.638140] Lustre: Unmounted lustre-client
[ 4005.844056] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 19:19:13 (1746141553)
[ 4022.346017] LustreError: 1968:0:(client.c:3262:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff8800a7db2680 x1830958022808832/t94489280516(94489280516) o101->lustre-MDT0000-mdc-ffff8800a9862000@192.168.204.129@tcp:12/10 lens 648/608 e 0 to 0 dl 1746141587 ref 2 fl Interpret:RPQU/204/0 rc 301/301 job:'multiop.0' uid:0 gid:0
[ 4025.847524] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4026.528050] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4030.175425] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 19:19:38 (1746141578)
[ 4030.791733] LustreError: lustre-OST0000-osc-ffff8800a9862000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4030.800267] Lustre: 1972:0:(llite_lib.c:4117:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.129@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 4030.814755] LustreError: 13939:0:(ldlm_resource.c:982:ldlm_resource_complain()) lustre-OST0000-osc-ffff8800a9862000: namespace resource [0x240000400:0x7382:0x0].0x0 (ffff88012fecef00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4030.821775] LustreError: 13939:0:(ldlm_resource.c:982:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4035.365981] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 19:19:43 (1746141583)
[ 4036.103045] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 4036.965177] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 19:19:45 (1746141585)
[ 4037.583177] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 4038.300062] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 19:19:46 (1746141586)
[ 4038.971296] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 4039.744239] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 19:19:47 (1746141587)
[ 4040.384404] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 4041.221670] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 19:19:49 (1746141589)
[ 4041.892904] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 4042.773252] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 19:19:50 (1746141590)
[ 4043.469582] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 4044.376832] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 19:19:52 (1746141592)
[ 4045.122392] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 4045.859636] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 19:19:54 (1746141594)
[ 4046.567056] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 4047.447415] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 19:19:55 (1746141595)
[ 4048.236561] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 4049.006832] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 19:19:57 (1746141597)
[ 4049.719782] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 4050.659124] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 19:19:58 (1746141598)
[ 4051.392082] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 4052.509651] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 19:20:00 (1746141600)
[ 4053.933085] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 4054.812692] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 19:20:02 (1746141602)
[ 4068.835656] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 19:20:16 (1746141616)
[ 4092.317001] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 19:20:40 (1746141640)
[ 4092.579383] Lustre: *** cfs_fail_loc=51b, val=3***
[ 4099.261721] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 19:20:47 (1746141647)
[ 4099.412103] Lustre: *** cfs_fail_loc=51b, val=4***
[ 4106.713471] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 19:20:54 (1746141654)
[ 4107.019015] Lustre: *** cfs_fail_loc=50f, val=3***
[ 4112.118455] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 19:21:00 (1746141660)
[ 4112.401063] Lustre: *** cfs_fail_loc=50f, val=4***
[ 4117.456960] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 19:21:05 (1746141665)
[ 4117.670237] Lustre: *** cfs_fail_loc=510, val=3***
[ 4122.720568] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 19:21:10 (1746141670)
[ 4122.904146] Lustre: *** cfs_fail_loc=51b, val=3***
[ 4184.643035] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 19:22:12 (1746141732)
[ 4184.926393] Lustre: *** cfs_fail_loc=51c, val=3***
[ 4247.424585] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 19:23:15 (1746141795)
[ 4247.680552] LustreError: 25158:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 4250.168794] LustreError: lustre-MDT0000-mdc-ffff8800a9862000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4250.178876] LustreError: 25174:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a9862000: inode [0x20000a042:0x8:0x0] mdc close failed: rc = -108
[ 4250.208051] LustreError: 25174:0:(ldlm_resource.c:982:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8800a9862000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff88012fecef00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4250.215276] LustreError: 25174:0:(ldlm_resource.c:982:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4251.720225] LustreError: 25158:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 4253.763789] LustreError: 25192:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 4256.088796] LustreError: lustre-MDT0000-mdc-ffff8800a9862000: operation ldlm_enqueue to node 192.168.204.129@tcp failed: rc = -107
[ 4256.095362] LustreError: Skipped 6 previous similar messages
[ 4256.101604] LustreError: lustre-MDT0000-mdc-ffff8800a9862000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4256.110484] LustreError: 25208:0:(ldlm_resource.c:982:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8800a9862000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8800b37d7c00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4256.121631] LustreError: 25208:0:(ldlm_resource.c:982:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4257.768621] LustreError: 25192:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 4264.236820] LustreError: lustre-MDT0000-mdc-ffff8800a9862000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4264.450726] LustreError: 25254:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 4266.988846] LustreError: lustre-MDT0000-mdc-ffff8800a9862000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4267.019509] LustreError: 25275:0:(file.c:6135:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4267.033007] LustreError: 25275:0:(file.c:6135:ll_inode_revalidate_fini()) Skipped 20 previous similar messages
[ 4268.466318] LustreError: 25254:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 4268.471343] LustreError: 25254:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a9862000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 4268.478740] LustreError: 25254:0:(file.c:247:ll_close_inode_openhandle()) Skipped 3 previous similar messages
[ 4273.524719] LustreError: 25334:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 4275.980586] LustreError: lustre-MDT0000-mdc-ffff8800a9862000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4275.992704] LustreError: 25350:0:(ldlm_resource.c:982:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8800a9862000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff88012fece100) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4276.004340] LustreError: 25350:0:(ldlm_resource.c:982:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4277.537348] LustreError: 25334:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 4283.993823] LustreError: lustre-MDT0000-mdc-ffff8800a9862000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4283.998780] LustreError: 25384:0:(ldlm_resource.c:982:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8800a9862000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff88012bf83700) refcount nonzero (2) after lock cleanup; forcing cleanup.
[ 4284.006251] LustreError: 25384:0:(ldlm_resource.c:982:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4284.177365] LustreError: 25398:0:(ldlm_flock.c:856:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 4286.541058] LustreError: lustre-MDT0000-mdc-ffff8800a9862000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4288.186325] LustreError: 25398:0:(ldlm_flock.c:856:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 4290.646471] LustreError: lustre-MDT0000-mdc-ffff8800a9862000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4290.690965] LustreError: 25449:0:(file.c:6135:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4290.713777] LustreError: 25449:0:(file.c:6135:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 4292.229413] LustreError: 25428:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a9862000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 4299.193023] LustreError: 25504:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 4299.199547] LustreError: 25504:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 4301.587487] LustreError: lustre-MDT0000-mdc-ffff8800a9862000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4303.211429] LustreError: 25504:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 4303.220059] LustreError: 25504:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 4303.226681] LustreError: 25504:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a9862000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 4308.861682] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 19:24:16 (1746141856)
[ 4326.142301] LustreError: 9112:0:(ldlm_lockd.c:2901:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 4326.147101] LustreError: 9112:0:(ldlm_lockd.c:2901:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 4330.155376] LustreError: 9112:0:(ldlm_lockd.c:2901:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 4330.161844] LustreError: 9112:0:(ldlm_lockd.c:2901:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 4333.653152] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 19:24:41 (1746141881)
[ 4375.759047] Lustre: lustre-MDT0000-mdc-ffff8800a9862000: Connection to lustre-MDT0000 (at 192.168.204.129@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4375.767532] Lustre: Skipped 28 previous similar messages
[ 4375.784061] Lustre: lustre-MDT0000-mdc-ffff8800a9862000: Connection restored to  (at 192.168.204.129@tcp)
[ 4375.789016] Lustre: Skipped 28 previous similar messages
[ 4380.032661] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 19:25:27 (1746141927)
[ 4436.438303] Lustre: 27905:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746141930/real 1746141930]  req@ffff88012e98b480 x1830958022932736/t0(0) o101->lustre-MDT0000-mdc-ffff8800a9862000@192.168.204.129@tcp:12/10 lens 576/1024 e 0 to 1 dl 1746141985 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'stat.0' uid:0 gid:0
[ 4436.463454] Lustre: 27905:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 13 previous similar messages
[ 4439.188943] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 19:26:27 (1746141987)
[ 4465.702852] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 19:26:53 (1746142013)
[ 4465.903122] Lustre: Mounted lustre-client
[ 4586.972539] LustreError: 29678:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 4586.976329] LustreError: 29678:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 4587.005937] Lustre: Unmounted lustre-client
[ 4589.766631] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 19:28:57 (1746142137)
[ 4590.024685] LustreError: 30351:0:(osc_request.c:2963:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 4000ms
[ 4592.107560] LustreError: lustre-OST0000-osc-ffff8800a9862000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4592.118145] LustreError: 30440:0:(ldlm_resource.c:982:ldlm_resource_complain()) lustre-OST0000-osc-ffff8800a9862000: namespace resource [0x240000400:0x73ad:0x0].0x0 (ffff8800a7565f00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4592.128189] LustreError: 30440:0:(ldlm_resource.c:982:ldlm_resource_complain()) Skipped 2 previous similar messages
[ 4594.038306] LustreError: 30351:0:(osc_request.c:2963:osc_build_rpc()) cfs_fail_timeout id 414 awake
[ 4594.047705] Lustre: 1970:0:(llite_lib.c:4117:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.129@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 4596.990932] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 19:29:05 (1746142145)
[ 4656.748813] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 19:30:04 (1746142204)
[ 4657.543644] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 4658.672299] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 19:30:06 (1746142206)
[ 4717.273048] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 4718.150259] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 19:31:06 (1746142266)
[ 4774.722016] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 19:32:02 (1746142322)
[ 4775.472974] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 4776.224841] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 19:32:04 (1746142324)
[ 4776.822474] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 4777.616717] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 19:32:05 (1746142325)
[ 4790.879783] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 19:32:19 (1746142339)
[ 4794.855182] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4814.066334] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 192.168.204.129@tcp) was lost; in progress operations using this service will fail
[ 4814.075033] LustreError: Skipped 2 previous similar messages
[ 4814.084714] Lustre: Evicted from MGS (at 192.168.204.129@tcp) after server handle changed from 0x2fbf3af67691d27e to 0x2fbf3af67691ec94
[ 4814.092148] Lustre: Skipped 2 previous similar messages
[ 4815.880259] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4816.619829] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4821.322036] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 19:32:49 (1746142369)
[ 4822.370607] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 4823.056646] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 19:32:51 (1746142371)
[ 4833.314717] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 19:33:01 (1746142381)
[ 4857.021862] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 19:33:25 (1746142405)
[ 4859.496416] LustreError: lustre-OST0000-osc-ffff8800a9862000: operation ost_setattr to node 192.168.204.129@tcp failed: rc = -19
[ 4859.502268] LustreError: Skipped 8 previous similar messages
[ 4881.516740] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4882.986106] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4965.292220] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4965.874915] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4984.436032] Lustre: lustre-MDT0000-mdc-ffff8800a9862000: Connection to lustre-MDT0000 (at 192.168.204.129@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4984.447615] Lustre: Skipped 10 previous similar messages
[ 4984.456118] Lustre: MGC192.168.204.129@tcp: Connection restored to 192.168.204.129@tcp (at 192.168.204.129@tcp)
[ 4984.463742] Lustre: Skipped 13 previous similar messages
[ 4987.607151] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4988.445886] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5023.796235] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 19:36:11 (1746142571)
[ 5052.918657] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5054.387347] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5210.091619] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 19:39:18 (1746142758)
[ 5310.964008] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 19:40:58 (1746142858)
[ 5311.667497] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 5312.487161] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 19:41:00 (1746142860)
[ 5313.546347] LustreError: lustre-MDT0000-mdc-ffff8800a9862000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5317.076574] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 19:41:05 (1746142865)
[ 5483.934311] Lustre: Evicted from lustre-OST0000_UUID (at 192.168.204.129@tcp) after server handle changed from 0x2fbf3af676a1aaf7 to 0x2fbf3af676a1abfa
[ 5483.942614] Lustre: Skipped 5 previous similar messages
[ 5483.949180] LustreError: lustre-OST0000-osc-ffff8800a9862000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5484.814388] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 19:43:52 (1746143032)
[ 5506.941376] Lustre: 1969:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746143035/real 1746143035]  req@ffff8801310b2d80 x1830958040677888/t0(0) o4->lustre-OST0000-osc-ffff8800a9862000@192.168.204.129@tcp:6/4 lens 4584/448 e 0 to 1 dl 1746143055 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0
[ 5506.957118] Lustre: 1969:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 13 previous similar messages
[ 5519.866413] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 19:44:28 (1746143068)
[ 5520.412108] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 5521.321088] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 19:44:29 (1746143069)
[ 5522.061065] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 5522.911476] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 19:44:30 (1746143070)
[ 5524.171599] Lustre: DEBUG MARKER: SKIP: recovery-small test_152 MDS Linux kernel does not support killable semaphore
[ 5525.058167] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 19:44:33 (1746143073)
[ 5557.090649] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 192.168.204.129@tcp) was lost; in progress operations using this service will fail
[ 5557.098834] LustreError: Skipped 5 previous similar messages
[ 5562.976705] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 19:45:11 (1746143111)
[ 5563.581804] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 5564.185858] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 19:45:12 (1746143112)
[ 5565.031119] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 5565.680696] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 19:45:13 (1746143113)
[ 5568.858318] LustreError: 18349:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup
[ 5568.861073] LustreError: 18349:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 5568.892324] Lustre: Unmounted lustre-client
[ 5569.348087] Lustre: Mounted lustre-client
[ 5570.436279] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5585.946687] Lustre: lustre-MDT0000-mdc-ffff8800a86ae000: Connection restored to 192.168.204.129@tcp (at 192.168.204.129@tcp)
[ 5585.951422] Lustre: Skipped 8 previous similar messages
[ 5587.602696] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: total 2265