[ 1903.385381] LustreError: lustre-OST0000-osc-ffff89add81b0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1903.406573] Lustre: 2370:0:(llite_lib.c:4215:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.132@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1903.421137] LustreError: 34732:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-OST0000-osc-ffff89add81b0000: namespace resource [0x240000400:0x45:0x0].0x0 (ffff89add0453d00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 1910.140788] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 15:16:57 (1753730217)
[ 1912.032816] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1913.762956] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 15:17:00 (1753730220)
[ 1915.458198] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1917.486988] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 15:17:04 (1753730224)
[ 1920.886042] LustreError: lustre-MDT0000-mdc-ffff89add81b0000: operation mds_reint to node 192.168.204.132@tcp failed: rc = -19
[ 1920.896166] LustreError: Skipped 1 previous similar message
[ 1939.945172] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 192.168.204.132@tcp) was lost; in progress operations using this service will fail
[ 1939.971631] Lustre: Evicted from MGS (at 192.168.204.132@tcp) after server handle changed from 0x3ba9b54e3c32430a to 0x3ba9b54e3c325793
[ 2048.444696] LustreError: lustre-MDT0000-mdc-ffff89add81b0000: operation ldlm_enqueue to node 192.168.204.132@tcp failed: rc = -19
[ 2048.453735] LustreError: Skipped 1 previous similar message
[ 2067.966945] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 192.168.204.132@tcp) was lost; in progress operations using this service will fail
[ 2067.999215] Lustre: Evicted from MGS (at 192.168.204.132@tcp) after server handle changed from 0x3ba9b54e3c325793 to 0x3ba9b54e3c342412
[ 2081.096622] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 15:19:48 (1753730388)
[ 2081.531596] Lustre: *** cfs_fail_loc=305, val=0***
[ 2081.533886] Lustre: Skipped 2 previous similar messages
[ 2124.300782] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 192.168.204.132@tcp) was lost; in progress operations using this service will fail
[ 2124.351789] Lustre: Evicted from MGS (at 192.168.204.132@tcp) after server handle changed from 0x3ba9b54e3c342412 to 0x3ba9b54e3c3444e2
[ 2135.028804] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2136.578901] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2144.966829] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 15:20:52 (1753730452)
[ 2160.107692] Lustre: lustre-MDT0000-mdc-ffff89add81b0000: Connection to lustre-MDT0000 (at 192.168.204.132@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2160.126736] Lustre: Skipped 12 previous similar messages
[ 2160.129610] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 192.168.204.132@tcp) was lost; in progress operations using this service will fail
[ 2160.165413] Lustre: Evicted from MGS (at 192.168.204.132@tcp) after server handle changed from 0x3ba9b54e3c3444e2 to 0x3ba9b54e3c344b95
[ 2160.177423] Lustre: MGC192.168.204.132@tcp: Connection restored to 192.168.204.132@tcp (at 192.168.204.132@tcp)
[ 2160.183783] Lustre: Skipped 15 previous similar messages
[ 2160.269617] LustreError: lustre-MDT0000-mdc-ffff89add81b0000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2179.420317] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 15:21:26 (1753730486)
[ 2209.832088] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 15:21:57 (1753730517)
[ 2223.453153] LustreError: lustre-MDT0000-mdc-ffff89add81b0000: operation ldlm_enqueue to node 192.168.204.132@tcp failed: rc = -19
[ 2223.458791] LustreError: Skipped 1 previous similar message
[ 2242.031555] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 192.168.204.132@tcp) was lost; in progress operations using this service will fail
[ 2242.058647] Lustre: Evicted from MGS (at 192.168.204.132@tcp) after server handle changed from 0x3ba9b54e3c344b95 to 0x3ba9b54e3c349beb
[ 2267.384042] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2269.595902] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2353.659593] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 192.168.204.132@tcp) was lost; in progress operations using this service will fail
[ 2353.727583] Lustre: Evicted from MGS (at 192.168.204.132@tcp) after server handle changed from 0x3ba9b54e3c349beb to 0x3ba9b54e3c367608
[ 2367.274945] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2369.631183] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2434.589488] LustreError: lustre-MDT0000-mdc-ffff89add81b0000: operation ldlm_enqueue to node 192.168.204.132@tcp failed: rc = -19
[ 2434.602674] LustreError: Skipped 1 previous similar message
[ 2451.937064] Lustre: 2370:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1753730744/real 1753730744]  req@ffff89adc8a45c00 x1838917529375488/t0(0) o400->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 224/224 e 0 to 1 dl 1753730760 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2451.953381] Lustre: 2370:0:(client.c:2453:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
[ 2451.960547] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 192.168.204.132@tcp) was lost; in progress operations using this service will fail
[ 2451.985860] Lustre: Evicted from MGS (at 192.168.204.132@tcp) after server handle changed from 0x3ba9b54e3c367608 to 0x3ba9b54e3c38a9ff
[ 2476.757432] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2478.630247] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2507.841025] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 15:26:54 (1753730814)
[ 2532.839777] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 192.168.204.132@tcp) was lost; in progress operations using this service will fail
[ 2532.861792] Lustre: Evicted from MGS (at 192.168.204.132@tcp) after server handle changed from 0x3ba9b54e3c38a9ff to 0x3ba9b54e3c397bb2
[ 2536.555059] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2565.854853] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2596.390148] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2644.562160] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2683.872712] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 192.168.204.132@tcp) was lost; in progress operations using this service will fail
[ 2683.876253] LustreError: Skipped 3 previous similar messages
[ 2692.256619] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2694.064830] Lustre: Evicted from MGS (at 192.168.204.132@tcp) after server handle changed from 0x3ba9b54e3c39bf58 to 0x3ba9b54e3c3a4006
[ 2694.071802] Lustre: Skipped 3 previous similar messages
[ 2720.859190] LustreError: lustre-MDT0000-mdc-ffff89add81b0000: operation mds_close to node 192.168.204.132@tcp failed: rc = -19
[ 2720.871578] LustreError: Skipped 5 previous similar messages
[ 2744.045675] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2777.388175] Lustre: lustre-MDT0000-mdc-ffff89add81b0000: Connection to lustre-MDT0000 (at 192.168.204.132@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2777.393547] Lustre: Skipped 9 previous similar messages
[ 2795.525856] Lustre: MGC192.168.204.132@tcp: Connection restored to 192.168.204.132@tcp (at 192.168.204.132@tcp)
[ 2795.533179] Lustre: Skipped 18 previous similar messages
[ 2829.414658] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 15:32:16 (1753731136)
[ 2874.948212] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2876.572496] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3203.985913] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3205.708722] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3505.473992] LustreError: lustre-OST0000-osc-ffff89add81b0000: operation ldlm_enqueue to node 192.168.204.132@tcp failed: rc = -19
[ 3505.478246] LustreError: Skipped 4 previous similar messages
[ 3505.496983] Lustre: lustre-OST0000-osc-ffff89add81b0000: Connection to lustre-OST0000 (at 192.168.204.132@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3505.517927] Lustre: Skipped 2 previous similar messages
[ 3538.652741] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3540.579408] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3798.919968] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 15:48:26 (1753732106)
[ 3815.904413] Lustre: 50248:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1753732108/real 1753732108]  req@ffff89adc8e77100 x1838917553932672/t0(0) o101->lustre-MDT0000-mdc-ffff89add81b0000@192.168.204.132@tcp:12/10 lens 576/1152 e 0 to 1 dl 1753732124 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3815.953664] Lustre: 50248:0:(client.c:2453:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
[ 3816.005504] Lustre: lustre-MDT0000-mdc-ffff89add81b0000: Connection restored to 192.168.204.132@tcp (at 192.168.204.132@tcp)
[ 3816.011966] Lustre: Skipped 1 previous similar message
[ 3824.037648] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 15:48:51 (1753732131)
[ 3848.251919] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 15:49:15 (1753732155)
[ 3873.876499] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 15:49:40 (1753732180)
[ 3874.218475] Lustre: Mounted lustre-client
[ 3905.001540] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 192.168.204.132@tcp) was lost; in progress operations using this service will fail
[ 3905.014534] LustreError: Skipped 2 previous similar messages
[ 3905.031708] Lustre: Evicted from MGS (at 192.168.204.132@tcp) after server handle changed from 0x3ba9b54e3c3bb69b to 0x3ba9b54e3c547ba6
[ 3905.042972] Lustre: Skipped 2 previous similar messages
[ 3906.017893] Lustre: 2372:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1753732198/real 1753732198]  req@ffff89adc8e74000 x1838917553950592/t0(0) o400->lustre-MDT0000-mdc-ffff89add81b0000@192.168.204.132@tcp:12/10 lens 224/224 e 0 to 1 dl 1753732214 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3906.058813] Lustre: 2372:0:(client.c:2453:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 3916.230567] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3917.718083] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3919.480588] LustreError: 53007:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff89add03fc800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 3919.496831] LustreError: 53007:0:(lov_obd.c:784:lov_cleanup()) Skipped 1 previous similar message
[ 3919.506542] LustreError: 53007:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 3919.511094] LustreError: 53007:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 3919.545981] Lustre: Unmounted lustre-client
[ 3924.828568] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 15:50:32 (1753732232)
[ 4069.344407] Lustre: 2370:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1753732361/real 1753732361]  req@ffff89adc8f0ea00 x1838917553999488/t0(0) o4->lustre-OST0000-osc-ffff89add81b0000@192.168.204.132@tcp:6/4 lens 488/448 e 0 to 1 dl 1753732377 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4069.385527] Lustre: 2370:0:(client.c:2453:ptlrpc_expire_one_request()) Skipped 90 previous similar messages
[ 4117.475142] Lustre: lustre-OST0000-osc-ffff89add81b0000: Connection to lustre-OST0000 (at 192.168.204.132@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4117.490357] Lustre: Skipped 15 previous similar messages
[ 4239.420581] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 15:55:46 (1753732546)
[ 4289.729640] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 15:56:36 (1753732596)
[ 4292.346974] LustreError: 55228:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff89add81b0000: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 4292.351498] LustreError: 55228:0:(lov_obd.c:784:lov_cleanup()) Skipped 1 previous similar message
[ 4292.364916] LustreError: 55228:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup
[ 4292.371516] LustreError: 55228:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 4292.416974] Lustre: Unmounted lustre-client
[ 4313.305728] Lustre: Mounted lustre-client
[ 4319.204375] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 15:57:06 (1753732626)
[ 4319.605801] LustreError: 56214:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4320.681390] LustreError: 56214:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4320.784589] Lustre: *** cfs_fail_loc=305, val=0***
[ 4343.755922] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 15:57:30 (1753732650)
[ 4344.512354] Lustre: Mounted lustre-client
[ 4346.332312] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4356.651550] LustreError: 56893:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 4356.677037] LustreError: 56893:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages
[ 4356.722666] Lustre: Unmounted lustre-client
[ 4364.244891] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 15:57:51 (1753732671)
[ 4464.206581] LustreError: lustre-MDT0000-mdc-ffff89add030d000: operation mds_reint to node 192.168.204.132@tcp failed: rc = -19
[ 4484.083591] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 192.168.204.132@tcp) was lost; in progress operations using this service will fail
[ 4484.116119] Lustre: Evicted from MGS (at 192.168.204.132@tcp) after server handle changed from 0x3ba9b54e3c548c23 to 0x3ba9b54e3c5741d9
[ 4484.137312] Lustre: MGC192.168.204.132@tcp: Connection restored to 192.168.204.132@tcp (at 192.168.204.132@tcp)
[ 4484.151903] Lustre: Skipped 23 previous similar messages
[ 4635.387771] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 16:02:22 (1753732942)
[ 4640.386496] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4653.039219] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 192.168.204.132@tcp) was lost; in progress operations using this service will fail
[ 4653.071699] Lustre: Evicted from MGS (at 192.168.204.132@tcp) after server handle changed from 0x3ba9b54e3c5741d9 to 0x3ba9b54e3c5c0a4d
[ 4653.199691] LustreError: lustre-MDT0000-mdc-ffff89add030d000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4667.965190] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 16:02:55 (1753732975)
[ 4668.382656] Lustre: Mounted lustre-client
[ 4675.812216] LustreError: lustre-OST0000-osc-ffff89add030d000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4685.792309] Lustre: 2369:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1753732978/real 1753732978]  req@ffff89adc9b31c00 x1838917558691968/t0(0) o103->lustre-OST0000-osc-ffff89adc7f88800@192.168.204.132@tcp:17/18 lens 328/224 e 0 to 1 dl 1753732994 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295
[ 4685.807801] Lustre: 2369:0:(client.c:2453:ptlrpc_expire_one_request()) Skipped 142 previous similar messages
[ 4687.132473] LustreError: 59952:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff89adc7f88800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 4687.145105] LustreError: 59952:0:(lov_obd.c:784:lov_cleanup()) Skipped 1 previous similar message
[ 4687.167828] LustreError: 59952:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 4687.174694] LustreError: 59952:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 4687.217793] Lustre: Unmounted lustre-client
[ 4694.920818] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 16:03:21 (1753733001)
[ 4702.037153] LustreError: lustre-MDT0000-mdc-ffff89add030d000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4702.074547] LustreError: 60596:0:(file.c:6198:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4710.263336] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 16:03:37 (1753733017)
[ 4710.477694] LustreError: 61294:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4714.646979] LustreError: lustre-MDT0000-mdc-ffff89add030d000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4714.660410] LustreError: 61317:0:(import.c:287:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4714.670039] LustreError: 61294:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=815
[ 4714.687144] LustreError: 61294:0:(import.c:709:ptlrpc_connect_import_locked()) already connecting
[ 4714.744386] LustreError: 61322:0:(file.c:6198:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4715.874175] LustreError: 61329:0:(file.c:6198:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4715.888700] LustreError: 61329:0:(file.c:6198:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4718.227206] LustreError: 61351:0:(file.c:6198:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4718.246147] LustreError: 61351:0:(file.c:6198:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4722.831312] LustreError: 61396:0:(file.c:6198:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4722.845125] LustreError: 61396:0:(file.c:6198:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4731.902333] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 16:03:59 (1753733039)
[ 4740.078510] Lustre: lustre-OST0000-osc-ffff89add030d000: Connection to lustre-OST0000 (at 192.168.204.132@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4740.090988] Lustre: Skipped 16 previous similar messages
[ 4765.383892] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4767.034390] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4777.523960] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 16:04:44 (1753733084)
[ 4811.544353] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4813.386211] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4825.224627] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 16:05:31 (1753733131)
[ 4886.232024] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4887.684103] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4896.513983] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 16:06:43 (1753733203)
[ 4932.797631] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4934.395380] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4940.256906] LustreError: 66970:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff89add030d000: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 4940.261787] LustreError: 66970:0:(lov_obd.c:784:lov_cleanup()) Skipped 1 previous similar message
[ 4940.275853] LustreError: 66970:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup
[ 4940.287894] LustreError: 66970:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 4940.345443] Lustre: Unmounted lustre-client
[ 4987.987833] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4990.389709] Lustre: Mounted lustre-client
[ 4996.143390] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 16:08:23 (1753733303)
[ 4999.086505] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5001.033639] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 16:08:27 (1753733307)
[ 5027.324994] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 16:08:54 (1753733334)
[ 5028.542194] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5029.925630] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 16:08:57 (1753733337)
[ 5031.249658] Lustre: *** cfs_fail_loc=805, val=0***
[ 5031.308231] Lustre: Mounted lustre-client
[ 5036.672568] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5057.010846] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 192.168.204.132@tcp) was lost; in progress operations using this service will fail
[ 5057.065144] Lustre: Evicted from MGS (at 192.168.204.132@tcp) after server handle changed from 0x3ba9b54e3c5c1433 to 0x3ba9b54e3c5c1743
[ 5062.810042] LustreError: 70317:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff89adca13e000: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 5062.813290] LustreError: 70317:0:(lov_obd.c:784:lov_cleanup()) Skipped 1 previous similar message
[ 5062.817180] LustreError: 70317:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 5062.822242] LustreError: 70317:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages
[ 5062.855086] Lustre: Unmounted lustre-client
[ 5069.911792] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 16:09:37 (1753733377)
[ 5091.783957] Lustre: MGC192.168.204.132@tcp: Connection restored to 192.168.204.132@tcp (at 192.168.204.132@tcp)
[ 5091.802860] Lustre: Skipped 15 previous similar messages
[ 5097.068952] LustreError: 2368:0:(client.c:3395:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff89adc9b32680 x1838917558772352/t90194313221(90194313221) o101->lustre-MDT0000-mdc-ffff89add801d800@192.168.204.132@tcp:12/10 lens 648/608 e 0 to 0 dl 1753733421 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5102.293108] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5104.064309] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5111.469712] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 16:10:18 (1753733418)
[ 5112.841893] LustreError: lustre-OST0000-osc-ffff89add801d800: operation ost_write to node 192.168.204.132@tcp failed: rc = -107
[ 5112.848597] LustreError: Skipped 3 previous similar messages
[ 5112.885135] LustreError: lustre-OST0000-osc-ffff89add801d800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5112.912448] Lustre: 2372:0:(llite_lib.c:4215:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.132@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5112.955597] LustreError: 72309:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-OST0000-osc-ffff89add801d800: namespace resource [0x240000400:0x4682:0x0].0x0 (ffff89adc4a43a00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5120.658911] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 16:10:27 (1753733427)
[ 5121.833740] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5123.279447] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 16:10:30 (1753733430)
[ 5124.468571] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5125.977331] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 16:10:33 (1753733433)
[ 5127.078667] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5128.640354] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 16:10:35 (1753733435)
[ 5129.881941] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5131.382756] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 16:10:38 (1753733438)
[ 5132.725901] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5134.189793] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 16:10:41 (1753733441)
[ 5135.854363] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5137.449761] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 16:10:44 (1753733444)
[ 5138.778336] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5140.276898] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 16:10:47 (1753733447)
[ 5141.718752] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5143.507551] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 16:10:50 (1753733450)
[ 5144.669769] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5145.822934] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 16:10:53 (1753733453)
[ 5147.121530] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5148.816126] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 16:10:55 (1753733455)
[ 5150.129062] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5151.863498] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 16:10:58 (1753733458)
[ 5154.601909] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5156.052881] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 16:11:03 (1753733463)
[ 5183.846846] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 16:11:31 (1753733491)
[ 5213.632278] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 16:12:00 (1753733520)
[ 5214.122329] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5224.444847] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 16:12:11 (1753733531)
[ 5224.699361] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5234.710959] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 16:12:21 (1753733541)
[ 5235.081322] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5242.616613] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 16:12:30 (1753733550)
[ 5242.924660] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5250.988601] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 16:12:38 (1753733558)
[ 5251.383987] Lustre: *** cfs_fail_loc=510, val=3***
[ 5260.086566] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 16:12:47 (1753733567)
[ 5260.488767] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5309.856908] Lustre: 2371:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1753733563/real 1753733563]  req@ffff89adc4988700 x1838917558813568/t0(0) o400->lustre-OST0001-osc-ffff89add801d800@192.168.204.132@tcp:28/4 lens 224/224 e 0 to 1 dl 1753733618 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5309.884784] Lustre: 2371:0:(client.c:2453:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
[ 5327.596533] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 16:13:54 (1753733634)
[ 5328.100147] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5386.720736] Lustre: lustre-OST0000-osc-ffff89add801d800: Connection to lustre-OST0000 (at 192.168.204.132@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5386.732337] Lustre: Skipped 10 previous similar messages
[ 5393.320722] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 16:15:00 (1753733700)
[ 5393.931236] LustreError: 82506:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5396.915837] LustreError: lustre-MDT0000-mdc-ffff89add801d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5396.936329] LustreError: 82521:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff89add801d800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff89adc3625800) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5398.008139] LustreError: 82506:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5400.089354] LustreError: 82536:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5403.139050] LustreError: lustre-MDT0000-mdc-ffff89add801d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5403.163505] LustreError: 82551:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff89add801d800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff89adc8790300) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5403.201163] LustreError: 82551:0:(ldlm_resource.c:981:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5404.184137] LustreError: 82536:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5411.078346] LustreError: lustre-MDT0000-mdc-ffff89add801d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5411.497624] LustreError: 82592:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5414.542756] LustreError: lustre-MDT0000-mdc-ffff89add801d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5414.626974] LustreError: 82611:0:(file.c:6198:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5414.641927] LustreError: 82611:0:(file.c:6198:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5415.600233] LustreError: 82592:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5415.619013] LustreError: 82592:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff89add801d800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5415.765804] LustreError: 82618:0:(file.c:6198:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5415.787098] LustreError: 82618:0:(file.c:6198:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5418.096930] LustreError: 82640:0:(file.c:6198:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5418.105350] LustreError: 82640:0:(file.c:6198:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5421.659049] LustreError: 82666:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5424.221337] LustreError: lustre-MDT0000-mdc-ffff89add801d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5424.235561] LustreError: 82681:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff89add801d800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff89adc4a43b00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5424.248981] LustreError: 82681:0:(ldlm_resource.c:981:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5425.728167] LustreError: 82666:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5432.702315] LustreError: lustre-MDT0000-mdc-ffff89add801d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5433.125511] LustreError: 82722:0:(ldlm_flock.c:856:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5435.984903] LustreError: lustre-MDT0000-mdc-ffff89add801d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5436.023593] LustreError: 82737:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff89add801d800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff89adc9b0b500) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5436.046874] LustreError: 82737:0:(ldlm_resource.c:981:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5436.059421] LustreError: 82742:0:(file.c:6198:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5436.059432] LustreError: 82742:0:(file.c:6198:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5437.200122] LustreError: 82722:0:(ldlm_flock.c:856:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5440.481031] LustreError: lustre-MDT0000-mdc-ffff89add801d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5441.785218] LustreError: 82761:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff89add801d800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5449.504477] LustreError: 82832:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5449.512475] LustreError: 82832:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5452.682666] LustreError: lustre-MDT0000-mdc-ffff89add801d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5452.741585] LustreError: 82854:0:(file.c:6198:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5452.749508] LustreError: 82854:0:(file.c:6198:ll_inode_revalidate_fini()) Skipped 28 previous similar messages
[ 5453.584140] LustreError: 82832:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5453.593118] LustreError: 82832:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5453.599744] LustreError: 82832:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff89add801d800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5463.214432] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 16:16:10 (1753733770)
[ 5482.892288] LustreError: 67800:0:(ldlm_lockd.c:2902:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5482.899359] LustreError: 67800:0:(ldlm_lockd.c:2902:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5486.970385] LustreError: 67800:0:(ldlm_lockd.c:2902:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5486.973155] LustreError: 67800:0:(ldlm_lockd.c:2902:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5493.262396] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 16:16:40 (1753733800)
[ 5544.533663] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 16:17:31 (1753733851)
[ 5609.814682] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 16:18:36 (1753733916)
[ 5643.973247] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 16:19:11 (1753733951)
[ 5644.568911] Lustre: Mounted lustre-client
[ 5767.621450] LustreError: 86549:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff89add83da800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 5767.638837] LustreError: 86549:0:(lov_obd.c:784:lov_cleanup()) Skipped 1 previous similar message
[ 5767.655726] LustreError: 86549:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 5767.670625] LustreError: 86549:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 5767.742183] Lustre: Unmounted lustre-client
[ 5775.013402] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 16:21:21 (1753734081)
[ 5775.684973] LustreError: 87165:0:(osc_request.c:2937:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 4000ms
[ 5778.807970] LustreError: lustre-OST0000-osc-ffff89add801d800: operation ost_statfs to node 192.168.204.132@tcp failed: rc = -107
[ 5778.813345] LustreError: Skipped 9 previous similar messages
[ 5778.822738] LustreError: lustre-OST0000-osc-ffff89add801d800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5778.839749] LustreError: 87252:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-OST0000-osc-ffff89add801d800: namespace resource [0x240000400:0x46ad:0x0].0x0 (ffff89adc3625900) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5778.862821] LustreError: 87252:0:(ldlm_resource.c:981:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5779.712357] LustreError: 87165:0:(osc_request.c:2937:osc_build_rpc()) cfs_fail_timeout id 414 awake
[ 5779.724743] Lustre: 2370:0:(llite_lib.c:4215:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.132@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5779.743140] Lustre: lustre-OST0000-osc-ffff89add801d800: Connection restored to 192.168.204.132@tcp (at 192.168.204.132@tcp)
[ 5779.756210] Lustre: Skipped 15 previous similar messages
[ 5786.358589] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 16:21:33 (1753734093)
[ 5850.481459] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 16:22:37 (1753734157)
[ 5852.146175] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 5854.173965] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 16:22:40 (1753734160)
[ 5911.520276] Lustre: 88825:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1753734164/real 1753734164]  req@ffff89adc4a2a680 x1838917558959360/t0(0) o101->lustre-MDT0000-mdc-ffff89add801d800@192.168.204.132@tcp:12/10 lens 648/66320 e 0 to 1 dl 1753734219 ref 2 fl Rpc:XPQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 5911.540380] Lustre: 88825:0:(client.c:2453:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
[ 5917.553203] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 5919.447385] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 16:23:46 (1753734226)
[ 5977.723650] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 16:24:44 (1753734284)
[ 5980.089286] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 5981.678773] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 16:24:48 (1753734288)
[ 5983.263351] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 5985.242822] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 16:24:52 (1753734292)
[ 6020.897463] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 16:25:27 (1753734327)
[ 6031.683383] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6057.968051] Lustre: lustre-MDT0000-mdc-ffff89add801d800: Connection to lustre-MDT0000 (at 192.168.204.132@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6057.969592] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 192.168.204.132@tcp) was lost; in progress operations using this service will fail
[ 6057.999626] Lustre: Skipped 16 previous similar messages
[ 6058.015038] LustreError: Skipped 2 previous similar messages
[ 6058.034070] Lustre: Evicted from MGS (at 192.168.204.132@tcp) after server handle changed from 0x3ba9b54e3c5c224f to 0x3ba9b54e3c5c3c9d
[ 6058.068708] Lustre: Skipped 2 previous similar messages
[ 6068.693904] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6070.591322] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6082.148271] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 16:26:28 (1753734388)
[ 6084.854813] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6087.083055] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 16:26:33 (1753734393)
[ 6098.939686] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 192.168.204.132@tcp) was lost; in progress operations using this service will fail
[ 6098.958718] Lustre: Evicted from MGS (at 192.168.204.132@tcp) after server handle changed from 0x3ba9b54e3c5c3c9d to 0x3ba9b54e3c5c3f44
[ 6113.480233] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 16:27:00 (1753734420)
[ 6154.111125] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 16:27:40 (1753734460)
[ 6202.390670] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 6205.597133] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6293.499148] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 192.168.204.132@tcp) was lost; in progress operations using this service will fail
[ 6293.524362] LustreError: Skipped 1 previous similar message
[ 6293.553318] Lustre: Evicted from MGS (at 192.168.204.132@tcp) after server handle changed from 0x3ba9b54e3c5c4517 to 0x3ba9b54e3c5c50fc
[ 6293.561705] Lustre: Skipped 1 previous similar message
[ 6307.369849] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6309.376910] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6358.454653] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6360.789456] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6477.088815] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 16:33:03 (1753734783)
[ 6490.940748] LustreError: lustre-OST0000-osc-ffff89add801d800: operation ost_setattr to node 192.168.204.132@tcp failed: rc = -107
[ 6490.952401] LustreError: Skipped 45 previous similar messages
[ 6535.234880] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 6539.711645] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6762.585219] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 16:37:49 (1753735069)
[ 6932.456595] Lustre: lustre-MDT0000-mdc-ffff89add801d800: Connection to lustre-MDT0000 (at 192.168.204.132@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6932.461311] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 192.168.204.132@tcp) was lost; in progress operations using this service will fail
[ 6932.467589] Lustre: Skipped 6 previous similar messages
[ 6932.487545] LustreError: Skipped 1 previous similar message
[ 6932.516939] Lustre: Evicted from MGS (at 192.168.204.132@tcp) after server handle changed from 0x3ba9b54e3c5c54fa to 0x3ba9b54e3c6bf7e2
[ 6932.529670] Lustre: Skipped 1 previous similar message
[ 6932.542383] Lustre: MGC192.168.204.132@tcp: Connection restored to 192.168.204.132@tcp (at 192.168.204.132@tcp)
[ 6932.554656] Lustre: Skipped 13 previous similar messages
[ 6958.143032] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 16:41:05 (1753735265)
[ 6959.811917] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 6962.180068] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 16:41:08 (1753735268)
[ 6963.917131] LustreError: lustre-MDT0000-mdc-ffff89add801d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6970.796092] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 16:41:18 (1753735278)
[ 7137.782209] LustreError: lustre-OST0000-osc-ffff89add801d800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7140.021987] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 16:44:06 (1753735446)
[ 7165.345107] Lustre: 2371:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1753735453/real 1753735453]  req@ffff89adc6e0b480 x1838917576651776/t0(0) o4->lustre-OST0000-osc-ffff89add801d800@192.168.204.132@tcp:6/4 lens 4584/448 e 0 to 1 dl 1753735473 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7165.361101] Lustre: 2371:0:(client.c:2453:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
[ 7182.647948] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 16:44:49 (1753735489)
[ 7184.151634] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7185.694590] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 16:44:52 (1753735492)
[ 7187.147953] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7188.602690] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 16:44:55 (1753735495)
[ 7221.458435] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 16:45:28 (1753735528)
[ 7257.579447] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 192.168.204.132@tcp) was lost; in progress operations using this service will fail
[ 7257.621571] Lustre: Evicted from MGS (at 192.168.204.132@tcp) after server handle changed from 0x3ba9b54e3c6bf7e2 to 0x3ba9b54e3c6c1fce
[ 7257.651143] Lustre: Skipped 1 previous similar message
[ 7273.731926] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 16:46:20 (1753735580)
[ 7275.074534] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7276.804479] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 16:46:23 (1753735583)
[ 7278.665197] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7280.506813] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 16:46:27 (1753735587)
[ 7290.147082] LustreError: 106051:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff89add801d800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 7290.162345] LustreError: 106051:0:(lov_obd.c:784:lov_cleanup()) Skipped 1 previous similar message
[ 7290.185778] LustreError: 106051:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup
[ 7290.188578] LustreError: 106051:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 7290.287055] Lustre: Unmounted lustre-client
[ 7291.535142] Lustre: Mounted lustre-client
[ 7294.512199] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7313.377053] Lustre: 2371:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1753735605/real 1753735605]  req@ffff89adc30d2680 x1838917576838272/t0(0) o400->MGC192.168.204.132@tcp@192.168.204.132@tcp:26/25 lens 224/224 e 0 to 1 dl 1753735621 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7336.003098] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: total 1809