[ 1462.688369] LustreError: lustre-OST0000-osc-ffff8800aaf9a000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1462.695107] Lustre: 1961:0:(llite_lib.c:4117:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.119@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1465.037893] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 23:51:57 (1746157917)
[ 1465.729547] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1466.267027] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 23:51:58 (1746157918)
[ 1466.978959] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1467.825033] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 23:51:59 (1746157919)
[ 1485.908593] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 192.168.202.119@tcp) was lost; in progress operations using this service will fail
[ 1485.933470] Lustre: Evicted from MGS (at 192.168.202.119@tcp) after server handle changed from 0xdc3f1aac150e1de0 to 0xdc3f1aac150e3d83
[ 1580.070637] LustreError: lustre-MDT0000-mdc-ffff8800aaf9a000: operation mds_reint to node 192.168.202.119@tcp failed: rc = -19
[ 1580.081351] LustreError: Skipped 3 previous similar messages
[ 1596.151103] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 192.168.202.119@tcp) was lost; in progress operations using this service will fail
[ 1596.193264] Lustre: Evicted from MGS (at 192.168.202.119@tcp) after server handle changed from 0xdc3f1aac150e3d83 to 0xdc3f1aac1512d0f2
[ 1600.511849] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 23:54:12 (1746158052)
[ 1600.747449] Lustre: *** cfs_fail_loc=305, val=0***
[ 1600.749292] Lustre: Skipped 4 previous similar messages
[ 1636.217555] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 192.168.202.119@tcp) was lost; in progress operations using this service will fail
[ 1636.239311] Lustre: Evicted from MGS (at 192.168.202.119@tcp) after server handle changed from 0xdc3f1aac1512d0f2 to 0xdc3f1aac1512e22c
[ 1639.917331] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1640.781369] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1645.394978] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 23:54:57 (1746158097)
[ 1651.268545] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 192.168.202.119@tcp) was lost; in progress operations using this service will fail
[ 1651.286684] Lustre: Evicted from MGS (at 192.168.202.119@tcp) after server handle changed from 0xdc3f1aac1512e22c to 0xdc3f1aac1512e6d2
[ 1651.329820] LustreError: lustre-MDT0000-mdc-ffff8800aaf9a000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 1662.698573] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 23:55:14 (1746158114)
[ 1669.217512] LustreError: lustre-OST0000-osc-ffff8800aaf9a000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1677.325511] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 23:55:29 (1746158129)
[ 1688.558183] LustreError: lustre-MDT0000-mdc-ffff8800aaf9a000: operation mds_close to node 192.168.202.119@tcp failed: rc = -19
[ 1706.323375] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 192.168.202.119@tcp) was lost; in progress operations using this service will fail
[ 1706.336509] Lustre: Evicted from MGS (at 192.168.202.119@tcp) after server handle changed from 0xdc3f1aac1512e6d2 to 0xdc3f1aac1513805b
[ 1709.415039] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1710.493895] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1716.194928] hrtimer: interrupt took 4375760 ns
[ 1773.243344] LustreError: lustre-MDT0000-mdc-ffff8800aaf9a000: operation ldlm_enqueue to node 192.168.202.119@tcp failed: rc = -19
[ 1773.249413] Lustre: lustre-MDT0000-mdc-ffff8800aaf9a000: Connection to lustre-MDT0000 (at 192.168.202.119@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 1773.249415] Lustre: Skipped 16 previous similar messages
[ 1773.258172] LustreError: Skipped 3 previous similar messages
[ 1791.527457] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 192.168.202.119@tcp) was lost; in progress operations using this service will fail
[ 1791.550932] Lustre: Evicted from MGS (at 192.168.202.119@tcp) after server handle changed from 0xdc3f1aac1513805b to 0xdc3f1aac15164728
[ 1791.558297] Lustre: MGC192.168.202.119@tcp: Connection restored to 192.168.202.119@tcp (at 192.168.202.119@tcp)
[ 1791.561594] Lustre: Skipped 21 previous similar messages
[ 1793.095979] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1793.874591] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1869.652974] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 192.168.202.119@tcp) was lost; in progress operations using this service will fail
[ 1869.667438] Lustre: Evicted from MGS (at 192.168.202.119@tcp) after server handle changed from 0xdc3f1aac15164728 to 0xdc3f1aac151a524f
[ 1876.192095] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1876.964170] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1900.646622] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 23:59:12 (1746158352)
[ 1903.087205] LustreError: lustre-MDT0000-mdc-ffff8800aaf9a000: operation ldlm_enqueue to node 192.168.202.119@tcp failed: rc = -19
[ 1903.092881] LustreError: Skipped 2 previous similar messages
[ 1919.925220] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 1938.055605] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 1939.762310] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 192.168.202.119@tcp) was lost; in progress operations using this service will fail
[ 1939.767684] LustreError: Skipped 1 previous similar message
[ 1939.790204] Lustre: Evicted from MGS (at 192.168.202.119@tcp) after server handle changed from 0xdc3f1aac151c004a to 0xdc3f1aac151c1eff
[ 1939.806068] Lustre: Skipped 1 previous similar message
[ 1960.930535] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 1988.657565] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2028.359897] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2069.927357] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 192.168.202.119@tcp) was lost; in progress operations using this service will fail
[ 2069.938339] LustreError: Skipped 3 previous similar messages
[ 2069.955366] Lustre: Evicted from MGS (at 192.168.202.119@tcp) after server handle changed from 0xdc3f1aac151e5939 to 0xdc3f1aac151f730e
[ 2069.960951] Lustre: Skipped 3 previous similar messages
[ 2072.128648] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2145.784507] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 00:03:17 (1746158597)
[ 2182.272169] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2183.511520] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 2487.415190] LustreError: lustre-OST0000-osc-ffff8800aaf9a000: operation ldlm_enqueue to node 192.168.202.119@tcp failed: rc = -19
[ 2487.419414] LustreError: Skipped 12 previous similar messages
[ 2487.422765] Lustre: lustre-OST0000-osc-ffff8800aaf9a000: Connection to lustre-OST0000 (at 192.168.202.119@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2487.438016] Lustre: Skipped 9 previous similar messages
[ 2514.308357] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2515.674494] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 2845.280650] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2846.218831] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3112.846778] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 00:19:24 (1746159564)
[ 3129.814135] Lustre: 22328:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746159566/real 1746159566]  req@ffff8800b4141180 x1830977826860032/t0(0) o101->lustre-MDT0000-mdc-ffff8800aaf9a000@192.168.202.119@tcp:12/10 lens 576/1024 e 0 to 1 dl 1746159582 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0
[ 3129.827561] Lustre: 22328:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
[ 3129.831347] Lustre: lustre-MDT0000-mdc-ffff8800aaf9a000: Connection to lustre-MDT0000 (at 192.168.202.119@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3129.841210] Lustre: Skipped 1 previous similar message
[ 3129.868114] Lustre: lustre-MDT0000-mdc-ffff8800aaf9a000: Connection restored to 192.168.202.119@tcp (at 192.168.202.119@tcp)
[ 3129.872520] Lustre: Skipped 18 previous similar messages
[ 3134.800050] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 00:19:46 (1746159586)
[ 3156.145600] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 00:20:08 (1746159608)
[ 3177.980925] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 00:20:29 (1746159629)
[ 3178.383358] Lustre: Mounted lustre-client
[ 3208.406084] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 192.168.202.119@tcp) was lost; in progress operations using this service will fail
[ 3208.412722] LustreError: Skipped 1 previous similar message
[ 3208.425343] Lustre: Evicted from MGS (at 192.168.202.119@tcp) after server handle changed from 0xdc3f1aac15209804 to 0xdc3f1aac1546fc35
[ 3208.436055] Lustre: Skipped 1 previous similar message
[ 3208.441910] Lustre: MGC192.168.202.119@tcp: Connection restored to  (at 192.168.202.119@tcp)
[ 3208.452084] Lustre: Skipped 2 previous similar messages
[ 3209.417082] Lustre: 1959:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746159646/real 1746159646]  req@ffff8800a89e6d80 x1830977826876288/t0(0) o400->lustre-MDT0000-mdc-ffff8800aaf9a000@192.168.202.119@tcp:12/10 lens 224/224 e 0 to 1 dl 1746159662 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0
[ 3209.428647] Lustre: 1959:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 3211.362315] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3212.386675] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3213.830082] LustreError: 25341:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 3213.837969] LustreError: 25341:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 3213.862247] Lustre: Unmounted lustre-client
[ 3217.557634] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 00:21:09 (1746159669)
[ 3369.667161] Lustre: 1959:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746159806/real 1746159806]  req@ffff8800a7d05880 x1830977826936320/t0(0) o4->lustre-OST0000-osc-ffff8800aaf9a000@192.168.202.119@tcp:6/4 lens 488/448 e 0 to 1 dl 1746159822 ref 2 fl Rpc:XQr/202/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0
[ 3369.696560] Lustre: 1959:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 124 previous similar messages
[ 3369.732880] Lustre: lustre-OST0000-osc-ffff8800aaf9a000: Connection restored to  (at 192.168.202.119@tcp)
[ 3369.732882] Lustre: Skipped 9 previous similar messages
[ 3515.914131] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 00:26:07 (1746159967)
[ 3561.837978] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 00:26:53 (1746160013)
[ 3563.574076] LustreError: 27902:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup
[ 3563.581905] LustreError: 27902:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 3563.619018] Lustre: Unmounted lustre-client
[ 3577.739570] Lustre: Mounted lustre-client
[ 3581.243857] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 00:27:13 (1746160033)
[ 3581.569445] LustreError: 28958:0:(mdc_locks.c:1333:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 3582.576317] LustreError: 28958:0:(mdc_locks.c:1333:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 3582.697895] Lustre: *** cfs_fail_loc=305, val=0***
[ 3603.390926] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 00:27:35 (1746160055)
[ 3603.671397] Lustre: Mounted lustre-client
[ 3604.543636] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 3614.581998] LustreError: 29701:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 3614.586851] LustreError: 29701:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages
[ 3614.609193] Lustre: Unmounted lustre-client
[ 3618.612207] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 00:27:50 (1746160070)
[ 3684.137628] LustreError: lustre-MDT0000-mdc-ffff8800b63e5800: operation mds_reint to node 192.168.202.119@tcp failed: rc = -19
[ 3684.145020] LustreError: Skipped 1 previous similar message
[ 3703.854896] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 192.168.202.119@tcp) was lost; in progress operations using this service will fail
[ 3703.874511] Lustre: Evicted from MGS (at 192.168.202.119@tcp) after server handle changed from 0xdc3f1aac15470cb9 to 0xdc3f1aac1549d0bc
[ 3703.882370] Lustre: MGC192.168.202.119@tcp: Connection restored to  (at 192.168.202.119@tcp)
[ 3703.884520] Lustre: Skipped 10 previous similar messages
[ 3804.760560] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 00:30:56 (1746160256)
[ 3808.028355] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 3819.033344] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 192.168.202.119@tcp) was lost; in progress operations using this service will fail
[ 3819.043725] Lustre: lustre-MDT0000-mdc-ffff8800b63e5800: Connection to lustre-MDT0000 (at 192.168.202.119@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3819.055028] Lustre: Skipped 24 previous similar messages
[ 3819.061135] Lustre: Evicted from MGS (at 192.168.202.119@tcp) after server handle changed from 0xdc3f1aac1549d0bc to 0xdc3f1aac154e80cc
[ 3819.133740] LustreError: lustre-MDT0000-mdc-ffff8800b63e5800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 3825.076905] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 00:31:17 (1746160277)
[ 3825.455658] Lustre: Mounted lustre-client
[ 3832.270850] LustreError: lustre-OST0000-osc-ffff8800b63e5800: operation ldlm_enqueue to node 192.168.202.119@tcp failed: rc = -107
[ 3832.286069] LustreError: lustre-OST0000-osc-ffff8800b63e5800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 3842.470849] Lustre: 1961:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746160279/real 1746160279]  req@ffff8800a2172680 x1830977831585664/t0(0) o103->lustre-OST0000-osc-ffff88012e595800@192.168.202.119@tcp:17/18 lens 328/224 e 0 to 1 dl 1746160295 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0
[ 3842.483336] Lustre: 1961:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 145 previous similar messages
[ 3843.039721] LustreError: 623:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 3843.042331] LustreError: 623:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 3843.078256] Lustre: Unmounted lustre-client
[ 3846.881114] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 00:31:38 (1746160298)
[ 3852.608479] LustreError: lustre-MDT0000-mdc-ffff8800b63e5800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 3852.621618] LustreError: 1358:0:(file.c:6135:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 3856.418626] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 00:31:48 (1746160308)
[ 3856.497901] LustreError: 2141:0:(recover.c:322:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 3860.002079] LustreError: lustre-MDT0000-mdc-ffff8800b63e5800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 3860.024907] LustreError: 2160:0:(import.c:287:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 3860.029967] LustreError: 2141:0:(recover.c:322:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=1475
[ 3860.041672] LustreError: 2141:0:(import.c:709:ptlrpc_connect_import_locked()) already connecting
[ 3860.054391] LustreError: 2166:0:(file.c:6135:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3861.115481] LustreError: 2172:0:(file.c:6135:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3861.124394] LustreError: 2172:0:(file.c:6135:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 3863.256965] LustreError: 2196:0:(file.c:6135:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3863.263548] LustreError: 2196:0:(file.c:6135:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 3867.629384] LustreError: 2244:0:(file.c:6135:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3867.646642] LustreError: 2244:0:(file.c:6135:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 3874.553887] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 00:32:06 (1746160326)
[ 3898.441247] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3899.465273] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3905.117964] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 00:32:37 (1746160357)
[ 3927.282487] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3928.048393] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3933.836310] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 00:33:06 (1746160386)
[ 3981.201507] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3981.896928] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3986.371791] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 00:33:58 (1746160438)
[ 4006.899166] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4007.557295] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4009.930131] LustreError: 8289:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup
[ 4009.932690] LustreError: 8289:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 4009.952177] Lustre: Unmounted lustre-client
[ 4035.942066] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4037.813532] Lustre: Mounted lustre-client
[ 4041.347289] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 00:34:53 (1746160493)
[ 4042.768554] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 4043.734868] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 00:34:55 (1746160495)
[ 4058.998793] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 00:35:11 (1746160511)
[ 4059.674396] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 4060.546166] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 00:35:12 (1746160512)
[ 4061.273418] Lustre: *** cfs_fail_loc=805, val=0***
[ 4061.293368] Lustre: Mounted lustre-client
[ 4063.864038] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4081.319113] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 192.168.202.119@tcp) was lost; in progress operations using this service will fail
[ 4081.337328] Lustre: Evicted from MGS (at 192.168.202.119@tcp) after server handle changed from 0xdc3f1aac154e8ac0 to 0xdc3f1aac154e8dc2
[ 4087.331507] LustreError: 1957:0:(ldlm_resource.c:982:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8800b6344000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff8800a20cff00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4087.949595] LustreError: 11831:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 4087.952523] LustreError: 11831:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages
[ 4087.979002] Lustre: Unmounted lustre-client
[ 4091.679040] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 00:35:43 (1746160543)
[ 4108.259442] LustreError: 1957:0:(client.c:3262:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff88009ec8ea00 x1830977831656192/t94489280517(94489280517) o101->lustre-MDT0000-mdc-ffff88012e590000@192.168.202.119@tcp:12/10 lens 648/608 e 0 to 0 dl 1746160577 ref 2 fl Interpret:RPQU/204/0 rc 301/301 job:'multiop.0' uid:0 gid:0
[ 4112.131316] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4112.916484] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4116.977377] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 00:36:09 (1746160569)
[ 4117.771539] LustreError: lustre-OST0000-osc-ffff88012e590000: operation ost_write to node 192.168.202.119@tcp failed: rc = -107
[ 4117.775146] LustreError: Skipped 2 previous similar messages
[ 4117.791772] LustreError: lustre-OST0000-osc-ffff88012e590000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4117.798630] Lustre: 1959:0:(llite_lib.c:4117:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.119@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 4117.813490] LustreError: 13991:0:(ldlm_resource.c:982:ldlm_resource_complain()) lustre-OST0000-osc-ffff88012e590000: namespace resource [0x240000400:0x6c82:0x0].0x0 (ffff88012dc29000) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4117.823558] LustreError: 13991:0:(ldlm_resource.c:982:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4122.099363] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 00:36:14 (1746160574)
[ 4122.740617] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 4123.620298] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 00:36:15 (1746160575)
[ 4124.654087] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 4125.727240] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 00:36:17 (1746160577)
[ 4126.360936] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 4127.178618] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 00:36:19 (1746160579)
[ 4127.901829] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 4128.719235] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 00:36:20 (1746160580)
[ 4129.503039] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 4130.447360] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 00:36:22 (1746160582)
[ 4131.213307] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 4132.058325] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 00:36:24 (1746160584)
[ 4132.626007] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 4133.476994] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 00:36:25 (1746160585)
[ 4134.108121] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 4134.783387] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 00:36:27 (1746160587)
[ 4135.440828] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 4136.258406] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 00:36:28 (1746160588)
[ 4136.931245] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 4137.874679] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 00:36:29 (1746160589)
[ 4138.510998] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 4139.298299] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 00:36:31 (1746160591)
[ 4140.549197] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 4141.505237] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 00:36:33 (1746160593)
[ 4155.381561] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 00:36:47 (1746160607)
[ 4180.386119] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 00:37:12 (1746160632)
[ 4180.690603] Lustre: *** cfs_fail_loc=51b, val=3***
[ 4187.330369] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 00:37:19 (1746160639)
[ 4187.485104] Lustre: *** cfs_fail_loc=51b, val=4***
[ 4193.889351] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 00:37:26 (1746160646)
[ 4194.007964] Lustre: *** cfs_fail_loc=50f, val=3***
[ 4197.516300] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 00:37:29 (1746160649)
[ 4197.583728] Lustre: *** cfs_fail_loc=50f, val=4***
[ 4200.985690] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 00:37:33 (1746160653)
[ 4201.191654] Lustre: *** cfs_fail_loc=510, val=3***
[ 4205.824166] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 00:37:38 (1746160658)
[ 4206.172804] Lustre: *** cfs_fail_loc=51b, val=3***
[ 4213.074550] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 00:37:45 (1746160665)
[ 4275.719392] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 00:38:47 (1746160727)
[ 4275.929854] LustreError: 25164:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 4278.402726] LustreError: lustre-MDT0000-mdc-ffff88012e590000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4278.420539] LustreError: 25180:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff88012e590000: inode [0x20000a042:0x8:0x0] mdc close failed: rc = -108
[ 4278.438385] LustreError: 25180:0:(ldlm_resource.c:982:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff88012e590000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8800a20ef100) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4278.446047] LustreError: 25180:0:(ldlm_resource.c:982:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4278.451395] LustreError: 25185:0:(file.c:6135:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4278.459100] LustreError: 25185:0:(file.c:6135:ll_inode_revalidate_fini()) Skipped 20 previous similar messages
[ 4279.947869] LustreError: 25164:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 4282.013849] LustreError: 25198:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 4284.410936] LustreError: lustre-MDT0000-mdc-ffff88012e590000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4284.419097] LustreError: 25214:0:(ldlm_resource.c:982:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff88012e590000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8800a20ef800) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4284.428549] LustreError: 25214:0:(ldlm_resource.c:982:ldlm_resource_complain()) Skipped 2 previous similar messages
[ 4286.033843] LustreError: 25198:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 4292.434958] LustreError: lustre-MDT0000-mdc-ffff88012e590000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4292.616459] LustreError: 25261:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 4294.965649] LustreError: lustre-MDT0000-mdc-ffff88012e590000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4294.995241] LustreError: 25282:0:(file.c:6135:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4296.629250] LustreError: 25261:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 4296.634660] LustreError: 25261:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff88012e590000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 4296.640663] LustreError: 25261:0:(file.c:247:ll_close_inode_openhandle()) Skipped 4 previous similar messages
[ 4301.451583] LustreError: 25341:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 4303.886614] LustreError: lustre-MDT0000-mdc-ffff88012e590000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4303.897645] LustreError: 25357:0:(ldlm_resource.c:982:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff88012e590000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8800a20ef800) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4303.903733] LustreError: 25357:0:(ldlm_resource.c:982:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4303.918559] Lustre: lustre-MDT0000-mdc-ffff88012e590000: Connection restored to 192.168.202.119@tcp (at 192.168.202.119@tcp)
[ 4303.924401] Lustre: Skipped 23 previous similar messages
[ 4305.460976] LustreError: 25341:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 4311.888757] LustreError: lustre-MDT0000-mdc-ffff88012e590000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4312.069488] LustreError: 25404:0:(ldlm_flock.c:856:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 4314.504883] LustreError: lustre-MDT0000-mdc-ffff88012e590000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4314.522725] LustreError: 25420:0:(ldlm_resource.c:982:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff88012e590000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8800a20ef600) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4314.529291] LustreError: 25420:0:(ldlm_resource.c:982:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4316.084793] LustreError: 25404:0:(ldlm_flock.c:856:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 4318.569935] LustreError: lustre-MDT0000-mdc-ffff88012e590000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4320.140580] LustreError: 25434:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff88012e590000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 4327.146753] LustreError: 25510:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 4327.153837] LustreError: 25510:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 4329.528310] LustreError: lustre-MDT0000-mdc-ffff88012e590000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4329.554198] LustreError: 25536:0:(file.c:6135:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4329.558089] LustreError: 25536:0:(file.c:6135:ll_inode_revalidate_fini()) Skipped 53 previous similar messages
[ 4331.170920] LustreError: 25510:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 4331.175653] LustreError: 25510:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 4331.180542] LustreError: 25510:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff88012e590000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 4335.991167] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 00:39:48 (1746160788)
[ 4352.867808] LustreError: 21587:0:(ldlm_lockd.c:2901:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 4352.871229] LustreError: 21587:0:(ldlm_lockd.c:2901:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 4356.878292] LustreError: 21587:0:(ldlm_lockd.c:2901:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 4356.881467] LustreError: 21587:0:(ldlm_lockd.c:2901:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 4359.687923] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 00:40:11 (1746160811)
[ 4404.011067] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 00:40:56 (1746160856)
[ 4459.812602] Lustre: 27907:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746160857/real 1746160857]  req@ffff8800a7cb2a00 x1830977831775872/t0(0) o101->lustre-MDT0000-mdc-ffff88012e590000@192.168.202.119@tcp:12/10 lens 576/1024 e 0 to 1 dl 1746160912 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'stat.0' uid:0 gid:0
[ 4459.825565] Lustre: 27907:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
[ 4459.828546] Lustre: lustre-MDT0000-mdc-ffff88012e590000: Connection to lustre-MDT0000 (at 192.168.202.119@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4459.834761] Lustre: Skipped 28 previous similar messages
[ 4462.460453] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 00:41:54 (1746160914)
[ 4488.820345] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 00:42:21 (1746160941)
[ 4489.087302] Lustre: Mounted lustre-client
[ 4610.266818] LustreError: 29681:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 4610.269819] LustreError: 29681:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 4610.290821] Lustre: Unmounted lustre-client
[ 4612.385653] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 00:44:24 (1746161064)
[ 4612.571109] LustreError: 30353:0:(osc_request.c:2943:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 4000ms
[ 4614.566929] LustreError: lustre-OST0000-osc-ffff88012e590000: operation ost_statfs to node 192.168.202.119@tcp failed: rc = -107
[ 4614.571277] LustreError: Skipped 9 previous similar messages
[ 4614.577292] LustreError: lustre-OST0000-osc-ffff88012e590000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4614.583073] LustreError: 30441:0:(ldlm_resource.c:982:ldlm_resource_complain()) lustre-OST0000-osc-ffff88012e590000: namespace resource [0x240000400:0x6cad:0x0].0x0 (ffff8800a20ef100) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4614.590722] LustreError: 30441:0:(ldlm_resource.c:982:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4616.585873] LustreError: 30353:0:(osc_request.c:2943:osc_build_rpc()) cfs_fail_timeout id 414 awake
[ 4616.591277] Lustre: 1961:0:(llite_lib.c:4117:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.119@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 4619.819956] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 00:44:31 (1746161071)
[ 4678.705533] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 00:45:31 (1746161131)
[ 4679.121855] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 4679.628055] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 00:45:31 (1746161131)
[ 4736.835535] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 4737.262815] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 00:46:29 (1746161189)
[ 4793.558413] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 00:47:25 (1746161245)
[ 4794.543139] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 4795.380348] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 00:47:27 (1746161247)
[ 4796.095050] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 4796.921457] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 00:47:29 (1746161249)
[ 4808.693938] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 00:47:40 (1746161260)
[ 4812.960552] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4827.766619] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 192.168.202.119@tcp) was lost; in progress operations using this service will fail
[ 4827.773791] LustreError: Skipped 2 previous similar messages
[ 4827.781192] Lustre: Evicted from MGS (at 192.168.202.119@tcp) after server handle changed from 0xdc3f1aac154e98c7 to 0xdc3f1aac154eb171
[ 4827.785543] Lustre: Skipped 2 previous similar messages
[ 4832.961931] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4833.663917] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4837.751404] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 00:48:10 (1746161290)
[ 4838.614416] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 4839.245926] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 00:48:11 (1746161291)
[ 4848.415196] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 00:48:20 (1746161300)
[ 4864.478675] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 00:48:36 (1746161316)
[ 4888.213900] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4889.243089] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4968.144712] Lustre: MGC192.168.202.119@tcp: Connection restored to 192.168.202.119@tcp (at 192.168.202.119@tcp)
[ 4968.151542] Lustre: Skipped 17 previous similar messages
[ 4970.602860] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4971.228129] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4992.351685] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4993.141653] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5024.565102] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 00:51:16 (1746161476)
[ 5057.785497] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5059.460448] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5217.574888] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 00:54:29 (1746161669)
[ 5303.840795] Lustre: lustre-MDT0000-mdc-ffff88012e590000: Connection to lustre-MDT0000 (at 192.168.202.119@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5303.849566] Lustre: Skipped 11 previous similar messages
[ 5326.123899] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 00:56:18 (1746161778)
[ 5326.724621] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 5327.459984] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 00:56:19 (1746161779)
[ 5328.358779] LustreError: lustre-MDT0000-mdc-ffff88012e590000: operation ldlm_enqueue to node 192.168.202.119@tcp failed: rc = -107
[ 5328.363400] LustreError: Skipped 339 previous similar messages
[ 5328.375593] LustreError: lustre-MDT0000-mdc-ffff88012e590000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5331.373073] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 00:56:23 (1746161783)
[ 5497.910774] Lustre: Evicted from lustre-OST0000_UUID (at 192.168.202.119@tcp) after server handle changed from 0xdc3f1aac155e7052 to 0xdc3f1aac155e7155
[ 5497.919593] Lustre: Skipped 5 previous similar messages
[ 5497.923043] LustreError: lustre-OST0000-osc-ffff88012e590000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5498.618024] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 00:59:10 (1746161950)
[ 5520.938816] Lustre: 1958:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746161953/real 1746161953]  req@ffff88012c758700 x1830977849533824/t0(0) o4->lustre-OST0000-osc-ffff88012e590000@192.168.202.119@tcp:6/4 lens 4584/448 e 0 to 1 dl 1746161973 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0
[ 5520.954002] Lustre: 1958:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 13 previous similar messages
[ 5534.157201] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 00:59:46 (1746161986)
[ 5534.899067] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 5535.860744] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 00:59:47 (1746161987)
[ 5536.658136] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 5537.718604] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 00:59:49 (1746161989)
[ 5538.995285] Lustre: DEBUG MARKER: SKIP: recovery-small test_152 MDS Linux kernel does not support killable semaphore
[ 5539.964026] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 00:59:51 (1746161991)
[ 5571.050138] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 192.168.202.119@tcp) was lost; in progress operations using this service will fail
[ 5571.057182] LustreError: Skipped 5 previous similar messages
[ 5571.070815] Lustre: MGC192.168.202.119@tcp: Connection restored to 192.168.202.119@tcp (at 192.168.202.119@tcp)
[ 5571.079334] Lustre: Skipped 7 previous similar messages
[ 5573.821025] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 01:00:26 (1746162026)
[ 5574.341464] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 5574.979346] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 01:00:27 (1746162027)
[ 5575.519313] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 5576.171511] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 01:00:28 (1746162028)
[ 5579.070464] LustreError: 18343:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup
[ 5579.072951] LustreError: 18343:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 5579.101000] Lustre: Unmounted lustre-client
[ 5579.613767] Lustre: Mounted lustre-client
[ 5580.840851] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5598.309276] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: total 2257