[ 1441.312499] LustreError: lustre-OST0000-osc-ffff8800aac63000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1441.319824] Lustre: 1961:0:(llite_lib.c:4117:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.115@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1443.568464] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 16:52:37 (1746132757)
[ 1444.269894] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1444.840696] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 16:52:39 (1746132759)
[ 1445.321958] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1445.853265] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 16:52:40 (1746132760)
[ 1465.200379] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 192.168.202.115@tcp) was lost; in progress operations using this service will fail
[ 1465.223672] Lustre: Evicted from MGS (at 192.168.202.115@tcp) after server handle changed from 0x8e6f033c3f309b33 to 0x8e6f033c3f30c82e
[ 1555.702301] LustreError: lustre-MDT0000-mdc-ffff8800aac63000: operation mds_reint to node 192.168.202.115@tcp failed: rc = -19
[ 1555.713091] LustreError: Skipped 2 previous similar messages
[ 1570.447235] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 192.168.202.115@tcp) was lost; in progress operations using this service will fail
[ 1570.458118] Lustre: Evicted from MGS (at 192.168.202.115@tcp) after server handle changed from 0x8e6f033c3f30c82e to 0x8e6f033c3f36bcfa
[ 1577.852194] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 16:54:51 (1746132891)
[ 1578.134398] Lustre: *** cfs_fail_loc=305, val=0***
[ 1578.136405] Lustre: Skipped 4 previous similar messages
[ 1615.953145] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 192.168.202.115@tcp) was lost; in progress operations using this service will fail
[ 1615.960888] Lustre: Evicted from MGS (at 192.168.202.115@tcp) after server handle changed from 0x8e6f033c3f36bcfa to 0x8e6f033c3f36c202
[ 1618.957611] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1619.474501] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1622.605217] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 16:55:36 (1746132936)
[ 1625.968012] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 192.168.202.115@tcp) was lost; in progress operations using this service will fail
[ 1625.974378] Lustre: Evicted from MGS (at 192.168.202.115@tcp) after server handle changed from 0x8e6f033c3f36c202 to 0x8e6f033c3f36c589
[ 1627.415539] LustreError: lustre-MDT0000-mdc-ffff8800aac63000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 1634.672741] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 16:55:48 (1746132948)
[ 1640.178077] LustreError: lustre-OST0000-osc-ffff8800aac63000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1647.096785] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 16:56:01 (1746132961)
[ 1657.973339] LustreError: lustre-MDT0000-mdc-ffff8800aac63000: operation ldlm_enqueue to node 192.168.202.115@tcp failed: rc = -19
[ 1657.981928] LustreError: Skipped 1 previous similar message
[ 1671.043737] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 192.168.202.115@tcp) was lost; in progress operations using this service will fail
[ 1671.069408] Lustre: Evicted from MGS (at 192.168.202.115@tcp) after server handle changed from 0x8e6f033c3f36c589 to 0x8e6f033c3f37c30a
[ 1677.410848] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1678.146667] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1756.190098] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 192.168.202.115@tcp) was lost; in progress operations using this service will fail
[ 1756.232192] Lustre: Evicted from MGS (at 192.168.202.115@tcp) after server handle changed from 0x8e6f033c3f37c30a to 0x8e6f033c3f3bb9ee
[ 1760.234078] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1761.031935] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1823.851057] LustreError: lustre-MDT0000-mdc-ffff8800aac63000: operation mds_reint to node 192.168.202.115@tcp failed: rc = -19
[ 1823.860093] LustreError: Skipped 1 previous similar message
[ 1841.313047] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 192.168.202.115@tcp) was lost; in progress operations using this service will fail
[ 1841.332871] Lustre: Evicted from MGS (at 192.168.202.115@tcp) after server handle changed from 0x8e6f033c3f3bb9ee to 0x8e6f033c3f3f73bc
[ 1842.762752] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1843.662404] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1868.040327] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 16:59:42 (1746133182)
[ 1887.231341] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 1906.403373] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 192.168.202.115@tcp) was lost; in progress operations using this service will fail
[ 1906.413942] LustreError: Skipped 1 previous similar message
[ 1906.446899] Lustre: Evicted from MGS (at 192.168.202.115@tcp) after server handle changed from 0x8e6f033c3f40ce27 to 0x8e6f033c3f40fe1d
[ 1906.454027] Lustre: Skipped 1 previous similar message
[ 1907.111222] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 1931.317255] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 1931.462661] Lustre: MGC192.168.202.115@tcp: Connection restored to 192.168.202.115@tcp (at 192.168.202.115@tcp)
[ 1931.470819] Lustre: Skipped 26 previous similar messages
[ 1942.703112] Lustre: lustre-MDT0000-mdc-ffff8800aac63000: Connection to lustre-MDT0000 (at 192.168.202.115@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 1942.714496] Lustre: Skipped 18 previous similar messages
[ 1959.457712] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 1996.308625] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2036.608178] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 192.168.202.115@tcp) was lost; in progress operations using this service will fail
[ 2036.613218] LustreError: Skipped 3 previous similar messages
[ 2036.618616] Lustre: Evicted from MGS (at 192.168.202.115@tcp) after server handle changed from 0x8e6f033c3f430a3d to 0x8e6f033c3f442729
[ 2036.624820] Lustre: Skipped 3 previous similar messages
[ 2038.760196] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2107.144233] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 17:03:41 (1746133421)
[ 2118.119481] LustreError: lustre-OST0000-osc-ffff8800aac63000: operation ldlm_enqueue to node 192.168.202.115@tcp failed: rc = -19
[ 2118.123441] LustreError: Skipped 7 previous similar messages
[ 2136.580373] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2137.383572] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 2474.034369] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2475.099806] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 2533.215640] hrtimer: interrupt took 5722329 ns
[ 2780.968626] LustreError: lustre-OST0000-osc-ffff8800aac63000: operation ldlm_enqueue to node 192.168.202.115@tcp failed: rc = -19
[ 2780.974144] LustreError: Skipped 1 previous similar message
[ 2780.977438] Lustre: lustre-OST0000-osc-ffff8800aac63000: Connection to lustre-OST0000 (at 192.168.202.115@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2780.987821] Lustre: Skipped 5 previous similar messages
[ 2803.039831] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2803.860093] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3075.097539] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 17:19:48 (1746134388)
[ 3092.327100] Lustre: 22321:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746134390/real 1746134390]  req@ffff8800a80e5c00 x1830951483489408/t0(0) o101->lustre-MDT0000-mdc-ffff8800aac63000@192.168.202.115@tcp:12/10 lens 576/1024 e 0 to 1 dl 1746134406 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0
[ 3092.346321] Lustre: 22321:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
[ 3092.375735] Lustre: lustre-MDT0000-mdc-ffff8800aac63000: Connection restored to 192.168.202.115@tcp (at 192.168.202.115@tcp)
[ 3092.380115] Lustre: Skipped 10 previous similar messages
[ 3096.620801] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 17:20:10 (1746134410)
[ 3117.859992] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 17:20:31 (1746134431)
[ 3139.502831] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 17:20:53 (1746134453)
[ 3139.876498] Lustre: Mounted lustre-client
[ 3169.909251] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 192.168.202.115@tcp) was lost; in progress operations using this service will fail
[ 3169.935627] LustreError: Skipped 1 previous similar message
[ 3169.947233] Lustre: Evicted from MGS (at 192.168.202.115@tcp) after server handle changed from 0x8e6f033c3f461941 to 0x8e6f033c3f72d078
[ 3169.951519] Lustre: Skipped 1 previous similar message
[ 3170.935108] Lustre: 1963:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746134469/real 1746134469]  req@ffff8800a8495500 x1830951483506816/t0(0) o400->lustre-MDT0000-mdc-ffff8800aac63000@192.168.202.115@tcp:12/10 lens 224/224 e 0 to 1 dl 1746134485 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0
[ 3170.961745] Lustre: 1963:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 3171.913596] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3172.792638] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3174.263332] LustreError: 25332:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 3174.266693] LustreError: 25332:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 3174.282759] Lustre: Unmounted lustre-client
[ 3177.138637] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 17:21:31 (1746134491)
[ 3329.496059] Lustre: 1963:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746134628/real 1746134628]  req@ffff8800a8dee680 x1830951483553792/t0(0) o4->lustre-OST0000-osc-ffff8800aac63000@192.168.202.115@tcp:6/4 lens 488/448 e 0 to 1 dl 1746134644 ref 2 fl Rpc:XQr/202/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0
[ 3329.507698] Lustre: 1963:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 58 previous similar messages
[ 3393.565887] Lustre: lustre-OST0000-osc-ffff8800aac63000: Connection to lustre-OST0000 (at 192.168.202.115@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3393.584011] Lustre: Skipped 17 previous similar messages
[ 3470.503809] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 17:26:24 (1746134784)
[ 3514.683537] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 17:27:08 (1746134828)
[ 3516.311837] LustreError: 27886:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup
[ 3516.314203] LustreError: 27886:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 3516.347904] Lustre: Unmounted lustre-client
[ 3529.421192] Lustre: Mounted lustre-client
[ 3533.012684] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 17:27:27 (1746134847)
[ 3533.217190] LustreError: 28943:0:(mdc_locks.c:1333:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 3534.231484] LustreError: 28943:0:(mdc_locks.c:1333:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 3534.334989] Lustre: *** cfs_fail_loc=305, val=0***
[ 3553.381725] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 17:27:47 (1746134867)
[ 3553.568397] Lustre: Mounted lustre-client
[ 3554.333307] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 3564.370923] LustreError: 29684:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 3564.377520] LustreError: 29684:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages
[ 3564.407812] Lustre: Unmounted lustre-client
[ 3568.378718] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 17:28:02 (1746134882)
[ 3633.219259] LustreError: lustre-MDT0000-mdc-ffff88009ed77000: operation mds_reint to node 192.168.202.115@tcp failed: rc = -19
[ 3633.229464] LustreError: Skipped 1 previous similar message
[ 3649.712776] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 192.168.202.115@tcp) was lost; in progress operations using this service will fail
[ 3649.725075] Lustre: Evicted from MGS (at 192.168.202.115@tcp) after server handle changed from 0x8e6f033c3f72e937 to 0x8e6f033c3f75ad80
[ 3744.214945] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 17:30:58 (1746135058)
[ 3747.157775] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 3754.935160] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 192.168.202.115@tcp) was lost; in progress operations using this service will fail
[ 3754.957646] Lustre: Evicted from MGS (at 192.168.202.115@tcp) after server handle changed from 0x8e6f033c3f75ad80 to 0x8e6f033c3f7a5c9b
[ 3754.966866] Lustre: MGC192.168.202.115@tcp: Connection restored to  (at 192.168.202.115@tcp)
[ 3754.975124] Lustre: Skipped 25 previous similar messages
[ 3755.023576] LustreError: lustre-MDT0000-mdc-ffff88009ed77000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 3765.534599] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 17:31:19 (1746135079)
[ 3765.972217] Lustre: Mounted lustre-client
[ 3772.842266] LustreError: lustre-OST0000-osc-ffff88009ed77000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 3782.759182] Lustre: 1963:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746135081/real 1746135081]  req@ffff8800a24aa680 x1830951488215808/t0(0) o103->lustre-OST0000-osc-ffff880131422000@192.168.202.115@tcp:17/18 lens 328/224 e 0 to 1 dl 1746135097 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0
[ 3782.778046] Lustre: 1963:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 95 previous similar messages
[ 3783.308499] LustreError: 592:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 3783.312068] LustreError: 592:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 3783.351986] Lustre: Unmounted lustre-client
[ 3786.395319] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 17:31:40 (1746135100)
[ 3792.089420] LustreError: lustre-MDT0000-mdc-ffff88009ed77000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 3792.096947] LustreError: 1326:0:(file.c:6135:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 3797.606333] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 17:31:50 (1746135110)
[ 3797.713705] LustreError: 2107:0:(recover.c:322:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 3802.351881] LustreError: lustre-MDT0000-mdc-ffff88009ed77000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 3802.359420] LustreError: 2132:0:(import.c:287:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 3802.368350] LustreError: 2107:0:(recover.c:322:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=355
[ 3802.377165] LustreError: 2107:0:(import.c:709:ptlrpc_connect_import_locked()) already connecting
[ 3802.387971] LustreError: 2137:0:(file.c:6135:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3803.444530] LustreError: 2144:0:(file.c:6135:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3803.454534] LustreError: 2144:0:(file.c:6135:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 3805.647851] LustreError: 2167:0:(file.c:6135:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3805.656382] LustreError: 2167:0:(file.c:6135:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 3810.137021] LustreError: 2215:0:(file.c:6135:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3810.144215] LustreError: 2215:0:(file.c:6135:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 3816.636529] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 17:32:10 (1746135130)
[ 3840.610298] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3841.809784] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3848.790320] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 17:32:42 (1746135162)
[ 3873.625367] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3874.256636] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3879.667589] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 17:33:13 (1746135193)
[ 3929.061372] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3929.899097] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3935.209579] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 17:34:09 (1746135249)
[ 3958.887893] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3959.892239] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3962.724097] LustreError: 8262:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup
[ 3962.726957] LustreError: 8262:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 3962.759222] Lustre: Unmounted lustre-client
[ 3997.747098] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3999.626671] Lustre: Mounted lustre-client
[ 4003.343704] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 17:35:17 (1746135317)
[ 4004.602449] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 4005.617043] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 17:35:19 (1746135319)
[ 4009.636871] Lustre: lustre-OST0000-osc-ffff88009ec38000: Connection to lustre-OST0000 (at 192.168.202.115@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4009.645133] Lustre: Skipped 18 previous similar messages
[ 4025.937780] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 17:35:39 (1746135339)
[ 4027.009753] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 4028.386844] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 17:35:41 (1746135341)
[ 4029.461885] Lustre: *** cfs_fail_loc=805, val=0***
[ 4029.504565] Lustre: Mounted lustre-client
[ 4033.773549] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4054.545347] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 192.168.202.115@tcp) was lost; in progress operations using this service will fail
[ 4054.565622] Lustre: Evicted from MGS (at 192.168.202.115@tcp) after server handle changed from 0x8e6f033c3f7a6730 to 0x8e6f033c3f7a6ab0
[ 4056.150614] LustreError: 11807:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 4056.158346] LustreError: 11807:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages
[ 4056.207437] Lustre: Unmounted lustre-client
[ 4062.205975] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 17:36:15 (1746135375)
[ 4089.121967] LustreError: 1960:0:(client.c:3262:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff8800a24a8700 x1830951488285824/t94489280517(94489280517) o101->lustre-MDT0000-mdc-ffff88009ec38000@192.168.202.115@tcp:12/10 lens 648/608 e 0 to 0 dl 1746135419 ref 2 fl Interpret:RPQU/204/0 rc 301/301 job:'multiop.0' uid:0 gid:0
[ 4091.334751] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4092.622528] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4094.217249] checkstat (13321) used greatest stack depth: 10624 bytes left
[ 4098.155408] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 17:36:52 (1746135412)
[ 4099.146392] LustreError: lustre-OST0000-osc-ffff88009ec38000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4099.157740] Lustre: 1961:0:(llite_lib.c:4117:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.115@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 4099.180236] LustreError: 13982:0:(ldlm_resource.c:982:ldlm_resource_complain()) lustre-OST0000-osc-ffff88009ec38000: namespace resource [0x240000400:0x7d82:0x0].0x0 (ffff8800b5c0a000) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4104.530001] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 17:36:58 (1746135418)
[ 4105.370869] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 4106.356473] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 17:37:00 (1746135420)
[ 4107.205234] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 4108.515505] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 17:37:02 (1746135422)
[ 4109.318973] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 4110.323630] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 17:37:04 (1746135424)
[ 4111.395448] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 4112.594866] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 17:37:06 (1746135426)
[ 4113.408496] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 4114.395590] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 17:37:08 (1746135428)
[ 4115.271851] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 4116.218625] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 17:37:10 (1746135430)
[ 4117.135375] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 4118.044342] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 17:37:11 (1746135431)
[ 4118.746783] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 4119.566644] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 17:37:13 (1746135433)
[ 4120.247486] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 4121.010273] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 17:37:15 (1746135435)
[ 4121.713714] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 4122.534581] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 17:37:16 (1746135436)
[ 4123.321400] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 4124.475702] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 17:37:18 (1746135438)
[ 4126.335993] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 4127.534647] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 17:37:21 (1746135441)
[ 4146.583830] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 17:37:40 (1746135460)
[ 4172.364354] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 17:38:06 (1746135486)
[ 4172.867514] Lustre: *** cfs_fail_loc=51b, val=3***
[ 4180.559328] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 17:38:14 (1746135494)
[ 4180.824295] Lustre: *** cfs_fail_loc=51b, val=4***
[ 4189.008353] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 17:38:22 (1746135502)
[ 4189.225576] Lustre: *** cfs_fail_loc=50f, val=3***
[ 4195.098942] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 17:38:28 (1746135508)
[ 4195.421625] Lustre: *** cfs_fail_loc=50f, val=4***
[ 4200.463025] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 17:38:34 (1746135514)
[ 4200.794594] Lustre: *** cfs_fail_loc=510, val=3***
[ 4205.354689] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 17:38:39 (1746135519)
[ 4205.673807] Lustre: *** cfs_fail_loc=51b, val=3***
[ 4213.140137] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 17:38:47 (1746135527)
[ 4275.971122] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 17:39:49 (1746135589)
[ 4276.186384] LustreError: 25170:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 4278.660405] LustreError: lustre-MDT0000-mdc-ffff88009ec38000: operation ldlm_enqueue to node 192.168.202.115@tcp failed: rc = -107
[ 4278.665214] LustreError: Skipped 4 previous similar messages
[ 4278.675961] LustreError: lustre-MDT0000-mdc-ffff88009ec38000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4278.685515] LustreError: 25186:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff88009ec38000: inode [0x20000a042:0xf:0x0] mdc close failed: rc = -108
[ 4278.693513] LustreError: 25186:0:(ldlm_resource.c:982:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff88009ec38000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8800a84cf300) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4280.198436] LustreError: 25170:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 4282.269030] LustreError: 25204:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 4284.785763] LustreError: lustre-MDT0000-mdc-ffff88009ec38000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4284.792919] LustreError: 25220:0:(ldlm_resource.c:982:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff88009ec38000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8800a84cf100) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4284.798949] LustreError: 25220:0:(ldlm_resource.c:982:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4286.274597] LustreError: 25204:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 4292.961694] LustreError: lustre-MDT0000-mdc-ffff88009ec38000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4293.201369] LustreError: 25267:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 4295.641266] LustreError: lustre-MDT0000-mdc-ffff88009ec38000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4295.676445] LustreError: 25288:0:(file.c:6135:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4295.681878] LustreError: 25288:0:(file.c:6135:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4297.217208] LustreError: 25267:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 4297.221783] LustreError: 25267:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff88009ec38000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 4302.211290] LustreError: 25347:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 4304.612793] LustreError: lustre-MDT0000-mdc-ffff88009ec38000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4304.622710] LustreError: 25363:0:(ldlm_resource.c:982:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff88009ec38000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8800b6d9de00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4304.629412] LustreError: 25363:0:(ldlm_resource.c:982:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4306.222445] LustreError: 25347:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 4312.800035] LustreError: lustre-MDT0000-mdc-ffff88009ec38000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4313.084100] LustreError: 25410:0:(ldlm_flock.c:856:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 4315.589612] LustreError: lustre-MDT0000-mdc-ffff88009ec38000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4315.601167] LustreError: 25426:0:(ldlm_resource.c:982:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff88009ec38000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8800b6d9d700) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4315.610333] LustreError: 25426:0:(ldlm_resource.c:982:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4317.099057] LustreError: 25410:0:(ldlm_flock.c:856:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 4319.659166] LustreError: lustre-MDT0000-mdc-ffff88009ec38000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4319.698773] LustreError: 25461:0:(file.c:6135:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4319.703211] LustreError: 25461:0:(file.c:6135:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 4321.186047] LustreError: 25440:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff88009ec38000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 4328.356578] LustreError: 25516:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 4328.360255] LustreError: 25516:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 4331.003258] LustreError: lustre-MDT0000-mdc-ffff88009ec38000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4332.419072] LustreError: 25516:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 4332.429378] LustreError: 25516:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 4332.440870] LustreError: 25516:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff88009ec38000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 4339.453895] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 17:40:53 (1746135653)
[ 4356.319299] Lustre: lustre-MDT0000-mdc-ffff88009ec38000: Connection restored to 192.168.202.115@tcp (at 192.168.202.115@tcp)
[ 4356.325351] Lustre: Skipped 26 previous similar messages
[ 4357.091449] LustreError: 9125:0:(ldlm_lockd.c:2901:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 4357.101543] LustreError: 9125:0:(ldlm_lockd.c:2901:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 4361.111042] LustreError: 9125:0:(ldlm_lockd.c:2901:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 4361.115008] LustreError: 9125:0:(ldlm_lockd.c:2901:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 4364.959850] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 17:41:18 (1746135678)
[ 4407.121030] Lustre: 27101:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746135680/real 1746135680]  req@ffff88012c526300 x1830951488402432/t0(0) o101->lustre-MDT0000-mdc-ffff88009ec38000@192.168.202.115@tcp:12/10 lens 576/1024 e 1 to 1 dl 1746135721 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'stat.0' uid:0 gid:0
[ 4407.134758] Lustre: 27101:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 9 previous similar messages
[ 4411.072788] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 17:42:04 (1746135724)
[ 4471.341018] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 17:43:05 (1746135785)
[ 4499.478289] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 17:43:33 (1746135813)
[ 4499.710886] Lustre: Mounted lustre-client
[ 4621.165130] LustreError: 29695:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 4621.169250] LustreError: 29695:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 4621.195424] Lustre: Unmounted lustre-client
[ 4624.634144] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 17:45:38 (1746135938)
[ 4625.071612] LustreError: 30370:0:(osc_request.c:2951:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 4000ms
[ 4627.511033] Lustre: lustre-OST0000-osc-ffff88009ec38000: Connection to lustre-OST0000 (at 192.168.202.115@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4627.523321] Lustre: Skipped 18 previous similar messages
[ 4627.533318] LustreError: lustre-OST0000-osc-ffff88009ec38000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4627.544240] LustreError: 30459:0:(ldlm_resource.c:982:ldlm_resource_complain()) lustre-OST0000-osc-ffff88009ec38000: namespace resource [0x240000400:0x7dad:0x0].0x0 (ffff88009eddd400) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4627.555052] LustreError: 30459:0:(ldlm_resource.c:982:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4629.090030] LustreError: 30370:0:(osc_request.c:2951:osc_build_rpc()) cfs_fail_timeout id 414 awake
[ 4629.101274] Lustre: 1964:0:(llite_lib.c:4117:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.115@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 4632.337159] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 17:45:46 (1746135946)
[ 4691.948870] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 17:46:46 (1746136006)
[ 4692.630739] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 4693.451712] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 17:46:47 (1746136007)
[ 4752.203748] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 4752.910128] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 17:47:47 (1746136067)
[ 4809.312709] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 17:48:43 (1746136123)
[ 4809.891462] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 4810.606772] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 17:48:44 (1746136124)
[ 4811.263719] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 4811.980332] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 17:48:46 (1746136126)
[ 4824.543114] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 17:48:58 (1746136138)
[ 4829.272877] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4848.834338] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 192.168.202.115@tcp) was lost; in progress operations using this service will fail
[ 4848.839036] LustreError: Skipped 2 previous similar messages
[ 4848.851762] Lustre: Evicted from MGS (at 192.168.202.115@tcp) after server handle changed from 0x8e6f033c3f7a750d to 0x8e6f033c3f7a9034
[ 4848.859197] Lustre: Skipped 2 previous similar messages
[ 4849.754814] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4850.293799] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4854.561022] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 17:49:28 (1746136168)
[ 4855.564190] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 4856.235186] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 17:49:30 (1746136170)
[ 4867.111432] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 17:49:41 (1746136181)
[ 4884.268153] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 17:49:58 (1746136198)
[ 4886.179837] LustreError: lustre-OST0000-osc-ffff88009ec38000: operation ost_setattr to node 192.168.202.115@tcp failed: rc = -19
[ 4886.184818] LustreError: Skipped 10 previous similar messages
[ 4910.954832] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4912.520780] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4989.381529] Lustre: MGC192.168.202.115@tcp: Connection restored to 192.168.202.115@tcp (at 192.168.202.115@tcp)
[ 4989.387253] Lustre: Skipped 12 previous similar messages
[ 4993.973714] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4994.651356] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5015.841825] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5016.367316] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5034.401785] Lustre: 1961:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746136294/real 1746136294]  req@ffff88009c0c6680 x1830951494958592/t0(0) o400->lustre-MDT0000-mdc-ffff88009ec38000@192.168.202.115@tcp:12/10 lens 224/224 e 0 to 1 dl 1746136349 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0
[ 5034.421512] Lustre: 1961:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 10 previous similar messages
[ 5043.704061] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 17:52:37 (1746136357)
[ 5068.352325] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5069.182789] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5251.627338] Lustre: lustre-MDT0000-mdc-ffff88009ec38000: Connection to lustre-MDT0000 (at 192.168.202.115@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5251.639342] Lustre: Skipped 10 previous similar messages
[ 5333.379871] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 17:57:27 (1746136647)
[ 5403.903633] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 192.168.202.115@tcp) was lost; in progress operations using this service will fail
[ 5403.910370] LustreError: Skipped 4 previous similar messages
[ 5403.917294] Lustre: Evicted from MGS (at 192.168.202.115@tcp) after server handle changed from 0x8e6f033c3f7aa71e to 0x8e6f033c3f8a497a
[ 5403.921640] Lustre: Skipped 4 previous similar messages
[ 5426.438031] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 17:59:00 (1746136740)
[ 5427.056698] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 5427.698834] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 17:59:01 (1746136741)
[ 5428.366334] LustreError: lustre-MDT0000-mdc-ffff88009ec38000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5430.623761] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 17:59:04 (1746136744)
[ 5596.451979] LustreError: lustre-OST0000-osc-ffff88009ec38000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5597.663361] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 18:01:51 (1746136911)
[ 5619.456939] Lustre: lustre-OST0000-osc-ffff88009ec38000: Connection restored to 192.168.202.115@tcp (at 192.168.202.115@tcp)
[ 5619.461469] Lustre: Skipped 8 previous similar messages
[ 5631.350624] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 18:02:25 (1746136945)
[ 5631.752685] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 5632.243578] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 18:02:26 (1746136946)
[ 5632.655984] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 5633.155357] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 18:02:27 (1746136947)
[ 5633.960730] Lustre: DEBUG MARKER: SKIP: recovery-small test_152 MDS Linux kernel does not support killable semaphore
[ 5634.543186] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 18:02:28 (1746136948)
[ 5672.851926] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 18:03:06 (1746136986)
[ 5673.498064] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 5674.252195] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 18:03:08 (1746136988)
[ 5674.869673] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 5675.543558] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 18:03:09 (1746136989)
[ 5678.101041] LustreError: 18432:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup
[ 5678.104512] LustreError: 18432:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 5678.138291] Lustre: Unmounted lustre-client
[ 5678.585446] Lustre: Mounted lustre-client
[ 5679.703424] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5695.613377] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: total 2259