[ 1680.214665] LustreError: lustre-OST0000-osc-ffff98a2cae2c000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1680.228447] Lustre: 2361:0:(llite_lib.c:4237:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.148@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1685.806652] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 15:09:52 (1764792592)
[ 1687.454831] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1689.091188] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 15:09:55 (1764792595)
[ 1690.618740] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1692.347901] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 15:09:58 (1764792598)
[ 1696.145171] LustreError: lustre-MDT0000-mdc-ffff98a2cae2c000: operation mds_reint to node 192.168.201.148@tcp failed: rc = -19
[ 1696.161051] LustreError: Skipped 1 previous similar message
[ 1715.171366] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 192.168.201.148@tcp) was lost; in progress operations using this service will fail
[ 1715.202445] Lustre: Evicted from MGS (at 192.168.201.148@tcp) after server handle changed from 0x50b7f38e58a71012 to 0x50b7f38e58a72a67
[ 1827.164619] LustreError: lustre-MDT0000-mdc-ffff98a2cae2c000: operation ldlm_enqueue to node 192.168.201.148@tcp failed: rc = -19
[ 1847.277193] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 192.168.201.148@tcp) was lost; in progress operations using this service will fail
[ 1847.316362] Lustre: Evicted from MGS (at 192.168.201.148@tcp) after server handle changed from 0x50b7f38e58a72a67 to 0x50b7f38e58a96446
[ 1875.057357] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 15:13:00 (1764792780)
[ 1875.862169] Lustre: *** cfs_fail_loc=305, val=0***
[ 1875.872108] Lustre: Skipped 4 previous similar messages
[ 1912.800317] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 192.168.201.148@tcp) was lost; in progress operations using this service will fail
[ 1921.976997] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1923.118746] Lustre: Evicted from MGS (at 192.168.201.148@tcp) after server handle changed from 0x50b7f38e58a96446 to 0x50b7f38e58a96be0
[ 1923.422102] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1931.740969] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 15:13:57 (1764792837)
[ 1948.648189] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 192.168.201.148@tcp) was lost; in progress operations using this service will fail
[ 1948.711265] Lustre: Evicted from MGS (at 192.168.201.148@tcp) after server handle changed from 0x50b7f38e58a96be0 to 0x50b7f38e58a96f52
[ 1948.796944] LustreError: lustre-MDT0000-mdc-ffff98a2cae2c000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 1968.367591] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 15:14:34 (1764792874)
[ 1984.493629] LustreError: lustre-OST0000-osc-ffff98a2cae2c000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2001.310422] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 15:15:06 (1764792906)
[ 2014.186922] LustreError: lustre-MDT0000-mdc-ffff98a2cae2c000: operation mds_reint to node 192.168.201.148@tcp failed: rc = -19
[ 2030.569063] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 192.168.201.148@tcp) was lost; in progress operations using this service will fail
[ 2030.587043] Lustre: Evicted from MGS (at 192.168.201.148@tcp) after server handle changed from 0x50b7f38e58a96f52 to 0x50b7f38e58a9c964
[ 2042.752882] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2045.016477] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2109.658663] LustreError: lustre-MDT0000-mdc-ffff98a2cae2c000: operation mds_reint to node 192.168.201.148@tcp failed: rc = -19
[ 2127.841585] Lustre: 2361:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764793019/real 1764793019]  req@ffff98a2c5057800 x1850517454401536/t0(0) o400->MGC192.168.201.148@tcp@192.168.201.148@tcp:26/25 lens 224/224 e 0 to 1 dl 1764793035 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2127.869182] Lustre: 2361:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
[ 2127.872659] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 192.168.201.148@tcp) was lost; in progress operations using this service will fail
[ 2127.912038] Lustre: Evicted from MGS (at 192.168.201.148@tcp) after server handle changed from 0x50b7f38e58a9c964 to 0x50b7f38e58abf589
[ 2127.917167] Lustre: MGC192.168.201.148@tcp: Connection restored to 192.168.201.148@tcp (at 192.168.201.148@tcp)
[ 2127.922969] Lustre: Skipped 18 previous similar messages
[ 2152.650950] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2155.933738] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2222.412015] Lustre: lustre-MDT0000-mdc-ffff98a2cae2c000: Connection to lustre-MDT0000 (at 192.168.201.148@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2222.453513] Lustre: Skipped 14 previous similar messages
[ 2240.480240] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 192.168.201.148@tcp) was lost; in progress operations using this service will fail
[ 2240.503130] Lustre: Evicted from MGS (at 192.168.201.148@tcp) after server handle changed from 0x50b7f38e58abf589 to 0x50b7f38e58ada012
[ 2262.777894] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2266.335153] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2300.798380] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 15:20:05 (1764793205)
[ 2306.875256] LustreError: lustre-MDT0000-mdc-ffff98a2cae2c000: operation mds_reint to node 192.168.201.148@tcp failed: rc = -19
[ 2306.900468] LustreError: Skipped 1 previous similar message
[ 2325.984262] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 192.168.201.148@tcp) was lost; in progress operations using this service will fail
[ 2333.709814] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2351.524931] Lustre: Evicted from MGS (at 192.168.201.148@tcp) after server handle changed from 0x50b7f38e58ada012 to 0x50b7f38e58ae48b3
[ 2352.189946] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2373.200759] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2399.915281] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2436.852649] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2476.513990] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 192.168.201.148@tcp) was lost; in progress operations using this service will fail
[ 2476.517788] LustreError: Skipped 3 previous similar messages
[ 2478.528550] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2522.596148] Lustre: Evicted from MGS (at 192.168.201.148@tcp) after server handle changed from 0x50b7f38e58b18f8d to 0x50b7f38e58b37ac8
[ 2522.600863] Lustre: Skipped 4 previous similar messages
[ 2546.864742] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 15:24:13 (1764793453)
[ 2577.716086] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2578.666457] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 2888.754409] LustreError: lustre-OST0000-osc-ffff98a2cae2c000: operation ldlm_enqueue to node 192.168.201.148@tcp failed: rc = -107
[ 2888.758821] LustreError: Skipped 7 previous similar messages
[ 2888.760500] Lustre: lustre-OST0000-osc-ffff98a2cae2c000: Connection to lustre-OST0000 (at 192.168.201.148@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2888.765297] Lustre: Skipped 6 previous similar messages
[ 2909.702876] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2910.500294] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3249.650349] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3250.612249] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3512.317866] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 15:40:19 (1764794419)
[ 3529.184164] Lustre: 50208:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764794420/real 1764794420]  req@ffff98a2c5b79880 x1850517507845248/t0(0) o101->lustre-MDT0000-mdc-ffff98a2cae2c000@192.168.201.148@tcp:12/10 lens 576/1152 e 0 to 1 dl 1764794436 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3529.197364] Lustre: 50208:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 3529.200652] Lustre: lustre-MDT0000-mdc-ffff98a2cae2c000: Connection to lustre-MDT0000 (at 192.168.201.148@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3529.206562] Lustre: Skipped 1 previous similar message
[ 3529.218522] Lustre: lustre-MDT0000-mdc-ffff98a2cae2c000: Connection restored to 192.168.201.148@tcp (at 192.168.201.148@tcp)
[ 3529.222536] Lustre: Skipped 15 previous similar messages
[ 3532.145197] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 15:40:38 (1764794438)
[ 3551.597942] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 15:40:58 (1764794458)
[ 3571.074219] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 15:41:17 (1764794477)
[ 3571.193546] Lustre: Mounted lustre-client
[ 3596.770311] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 192.168.201.148@tcp) was lost; in progress operations using this service will fail
[ 3596.773993] LustreError: Skipped 1 previous similar message
[ 3596.778410] Lustre: Evicted from MGS (at 192.168.201.148@tcp) after server handle changed from 0x50b7f38e58b37ac8 to 0x50b7f38e58eb037b
[ 3599.529329] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3600.043192] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3601.368620] LustreError: 52962:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff98a2c30e9000: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 3601.371543] LustreError: 52962:0:(lov_obd.c:783:lov_cleanup()) Skipped 1 previous similar message
[ 3601.374378] LustreError: 52962:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 3601.376106] LustreError: 52962:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 3601.390353] Lustre: Unmounted lustre-client
[ 3603.478152] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 15:41:50 (1764794510)
[ 3608.032334] Lustre: 2364:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764794498/real 1764794498]  req@ffff98a2c577dc00 x1850517507869696/t0(0) o400->lustre-MDT0000-mdc-ffff98a2cae2c000@192.168.201.148@tcp:12/10 lens 224/224 e 0 to 1 dl 1764794514 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3608.053971] Lustre: 2364:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 3637.731879] Lustre: lustre-OST0000-osc-ffff98a2cae2c000: Connection restored to 192.168.201.148@tcp (at 192.168.201.148@tcp)
[ 3637.736063] Lustre: Skipped 5 previous similar messages
[ 3765.728150] Lustre: 2361:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764794657/real 1764794657]  req@ffff98a2c5b7ad80 x1850517507897984/t0(0) o4->lustre-OST0000-osc-ffff98a2cae2c000@192.168.201.148@tcp:6/4 lens 488/448 e 0 to 1 dl 1764794673 ref 2 fl Rpc:XQr/602/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 3765.769173] Lustre: 2361:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 85 previous similar messages
[ 3798.511394] Lustre: lustre-OST0000-osc-ffff98a2cae2c000: Connection restored to 192.168.201.148@tcp (at 192.168.201.148@tcp)
[ 3798.516437] Lustre: Skipped 9 previous similar messages
[ 3895.993122] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 15:46:41 (1764794801)
[ 3942.780078] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 15:47:29 (1764794849)
[ 3945.047595] LustreError: 55122:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff98a2cae2c000: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 3945.066092] LustreError: 55122:0:(lov_obd.c:783:lov_cleanup()) Skipped 1 previous similar message
[ 3945.082488] LustreError: 55122:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup
[ 3945.088305] LustreError: 55122:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 3945.151960] Lustre: Unmounted lustre-client
[ 3962.976950] Lustre: Mounted lustre-client
[ 3968.455163] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 15:47:54 (1764794874)
[ 3968.790588] LustreError: 56107:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 3969.843513] LustreError: 56107:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 3969.945607] Lustre: *** cfs_fail_loc=305, val=0***
[ 3992.468991] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 15:48:18 (1764794898)
[ 3992.904947] Lustre: Mounted lustre-client
[ 3994.535937] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4004.784995] LustreError: 56786:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 4004.796098] LustreError: 56786:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages
[ 4004.836386] Lustre: Unmounted lustre-client
[ 4011.483845] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 15:48:37 (1764794917)
[ 4080.800926] LustreError: lustre-MDT0000-mdc-ffff98a2c5b11000: operation mds_reint to node 192.168.201.148@tcp failed: rc = -19
[ 4080.806936] LustreError: Skipped 1 previous similar message
[ 4095.457835] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 192.168.201.148@tcp) was lost; in progress operations using this service will fail
[ 4095.464487] Lustre: Evicted from MGS (at 192.168.201.148@tcp) after server handle changed from 0x50b7f38e58eb199a to 0x50b7f38e58edf7c8
[ 4100.695948] Lustre: lustre-MDT0000-mdc-ffff98a2c5b11000: Connection restored to 192.168.201.148@tcp (at 192.168.201.148@tcp)
[ 4100.701710] Lustre: Skipped 8 previous similar messages
[ 4211.838970] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 15:51:58 (1764795118)
[ 4216.033708] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4229.107859] Lustre: lustre-MDT0000-mdc-ffff98a2c5b11000: Connection to lustre-MDT0000 (at 192.168.201.148@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4229.135074] Lustre: Skipped 24 previous similar messages
[ 4229.142388] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 192.168.201.148@tcp) was lost; in progress operations using this service will fail
[ 4229.186738] Lustre: Evicted from MGS (at 192.168.201.148@tcp) after server handle changed from 0x50b7f38e58edf7c8 to 0x50b7f38e58f2787c
[ 4229.233487] LustreError: lustre-MDT0000-mdc-ffff98a2c5b11000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4246.223071] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 15:52:31 (1764795151)
[ 4246.994277] Lustre: Mounted lustre-client
[ 4254.551333] LustreError: lustre-OST0000-osc-ffff98a2c5b11000: operation ldlm_enqueue to node 192.168.201.148@tcp failed: rc = -107
[ 4254.587685] LustreError: lustre-OST0000-osc-ffff98a2c5b11000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4264.416422] Lustre: 2362:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764795155/real 1764795155]  req@ffff98a2c8515f80 x1850517512503552/t0(0) o103->lustre-OST0000-osc-ffff98a2cae2f800@192.168.201.148@tcp:17/18 lens 328/224 e 0 to 1 dl 1764795171 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295
[ 4264.514231] Lustre: 2362:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 66 previous similar messages
[ 4266.697088] LustreError: 59830:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff98a2cae2f800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 4266.720789] LustreError: 59830:0:(lov_obd.c:783:lov_cleanup()) Skipped 1 previous similar message
[ 4266.822055] LustreError: 59830:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 4266.825365] LustreError: 59830:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 4266.847750] Lustre: Unmounted lustre-client
[ 4274.673490] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 15:53:00 (1764795180)
[ 4281.597573] LustreError: lustre-MDT0000-mdc-ffff98a2c5b11000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4281.620761] LustreError: 60473:0:(file.c:6118:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4281.634302] LustreError: 60557:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff98a2c5b11000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff98a2ca908700) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4288.591620] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 15:53:14 (1764795194)
[ 4288.775266] LustreError: 61176:0:(recover.c:330:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4294.113503] LustreError: 61176:0:(recover.c:330:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=0
[ 4294.122080] LustreError: 61176:0:(import.c:710:ptlrpc_connect_import_locked()) already connected
[ 4294.505966] LustreError: lustre-MDT0000-mdc-ffff98a2c5b11000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4294.512861] LustreError: 61194:0:(import.c:291:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4294.544834] LustreError: 61199:0:(file.c:6118:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4295.621978] LustreError: 61205:0:(file.c:6118:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4295.634657] LustreError: 61205:0:(file.c:6118:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4297.916483] LustreError: 61228:0:(file.c:6118:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4297.938714] LustreError: 61228:0:(file.c:6118:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4302.557707] LustreError: 61273:0:(file.c:6118:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4302.566085] LustreError: 61273:0:(file.c:6118:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4311.651762] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 15:53:37 (1764795217)
[ 4344.696511] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4346.358589] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4356.755593] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 15:54:22 (1764795262)
[ 4390.817420] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4392.820767] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4402.983401] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 15:55:08 (1764795308)
[ 4464.372267] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4466.317307] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4474.611620] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 15:56:20 (1764795380)
[ 4510.295932] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4511.797432] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4516.933301] LustreError: 66848:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff98a2c5b11000: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 4516.940643] LustreError: 66848:0:(lov_obd.c:783:lov_cleanup()) Skipped 1 previous similar message
[ 4516.951848] LustreError: 66848:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup
[ 4516.954696] LustreError: 66848:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 4516.997139] Lustre: Unmounted lustre-client
[ 4567.125844] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4570.425240] Lustre: Mounted lustre-client
[ 4577.680538] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 15:58:03 (1764795483)
[ 4580.557635] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 4582.386468] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 15:58:08 (1764795488)
[ 4608.869260] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 15:58:35 (1764795515)
[ 4610.366767] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 4611.996039] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 15:58:37 (1764795517)
[ 4613.399484] Lustre: *** cfs_fail_loc=805, val=0***
[ 4613.443136] Lustre: Mounted lustre-client
[ 4618.922182] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4639.209602] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 192.168.201.148@tcp) was lost; in progress operations using this service will fail
[ 4639.236237] Lustre: Evicted from MGS (at 192.168.201.148@tcp) after server handle changed from 0x50b7f38e58f2825b to 0x50b7f38e58f28484
[ 4650.671452] LustreError: 70195:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff98a2c8470800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 4650.688629] LustreError: 70195:0:(lov_obd.c:783:lov_cleanup()) Skipped 1 previous similar message
[ 4650.720654] LustreError: 70195:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 4650.737201] LustreError: 70195:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages
[ 4650.810420] Lustre: Unmounted lustre-client
[ 4662.839477] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 15:59:28 (1764795568)
[ 4705.848594] LustreError: 2360:0:(client.c:3408:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff98a2c64c8000 x1850517512582912/t85899345924(85899345924) o101->lustre-MDT0000-mdc-ffff98a2c3023800@192.168.201.148@tcp:12/10 lens 664/608 e 0 to 0 dl 1764795629 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 4706.013056] Lustre: lustre-MDT0000-mdc-ffff98a2c3023800: Connection restored to 192.168.201.148@tcp (at 192.168.201.148@tcp)
[ 4706.039740] Lustre: Skipped 16 previous similar messages
[ 4710.570368] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4712.503094] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4722.632146] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 16:00:28 (1764795628)
[ 4724.457792] LustreError: lustre-OST0000-osc-ffff98a2c3023800: operation ost_write to node 192.168.201.148@tcp failed: rc = -107
[ 4724.469756] LustreError: Skipped 2 previous similar messages
[ 4724.499369] LustreError: lustre-OST0000-osc-ffff98a2c3023800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4724.518577] Lustre: 2364:0:(llite_lib.c:4237:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.148@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 4724.577881] LustreError: 72192:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-OST0000-osc-ffff98a2c3023800: namespace resource [0x240000400:0x7ce2:0x0].0x0 (ffff98a2c8b52400) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4732.414969] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 16:00:38 (1764795638)
[ 4733.598470] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 4734.915469] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 16:00:41 (1764795641)
[ 4736.227455] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 4737.690550] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 16:00:43 (1764795643)
[ 4738.927752] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 4740.534668] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 16:00:46 (1764795646)
[ 4741.932344] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 4743.968619] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 16:00:49 (1764795649)
[ 4745.409340] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 4747.162497] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 16:00:53 (1764795653)
[ 4748.714467] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 4750.324402] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 16:00:56 (1764795656)
[ 4751.704660] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 4753.406917] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 16:00:59 (1764795659)
[ 4754.824991] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 4756.698825] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 16:01:02 (1764795662)
[ 4758.197320] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 4760.249996] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 16:01:06 (1764795666)
[ 4761.956708] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 4763.508985] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 16:01:09 (1764795669)
[ 4764.979415] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 4766.750545] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 16:01:12 (1764795672)
[ 4769.637638] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 4771.643554] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 16:01:17 (1764795677)
[ 4800.422767] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 16:01:46 (1764795706)
[ 4829.205720] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 16:02:15 (1764795735)
[ 4829.636530] Lustre: *** cfs_fail_loc=51b, val=3***
[ 4838.983777] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 16:02:25 (1764795745)
[ 4839.339506] Lustre: *** cfs_fail_loc=51b, val=4***
[ 4848.820803] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 16:02:34 (1764795754)
[ 4849.168331] Lustre: *** cfs_fail_loc=50f, val=3***
[ 4856.514990] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 16:02:42 (1764795762)
[ 4856.695700] Lustre: *** cfs_fail_loc=50f, val=4***
[ 4864.134758] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 16:02:50 (1764795770)
[ 4864.485022] Lustre: *** cfs_fail_loc=510, val=3***
[ 4872.605943] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 16:02:58 (1764795778)
[ 4872.878514] Lustre: *** cfs_fail_loc=51b, val=3***
[ 4882.180337] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 16:03:08 (1764795788)
[ 4882.460898] Lustre: *** cfs_fail_loc=51c, val=3***
[ 4930.016342] Lustre: 2364:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764795782/real 1764795782]  req@ffff98a2c82c4e00 x1850517512623488/t0(0) o400->lustre-OST0001-osc-ffff98a2c3023800@192.168.201.148@tcp:28/4 lens 224/224 e 0 to 1 dl 1764795837 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4930.047831] Lustre: 2364:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 17 previous similar messages
[ 4941.793139] Lustre: lustre-OST0000-osc-ffff98a2c3023800: Connection to lustre-OST0000 (at 192.168.201.148@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4941.820584] Lustre: Skipped 17 previous similar messages
[ 4948.888939] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 16:04:14 (1764795854)
[ 4949.488762] LustreError: 82392:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 4952.264981] LustreError: lustre-MDT0000-mdc-ffff98a2c3023800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4952.300236] LustreError: 82406:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff98a2c3023800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff98a2c9c02e00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4953.504167] LustreError: 82392:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 4955.686653] LustreError: 82422:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 4959.078783] LustreError: lustre-MDT0000-mdc-ffff98a2c3023800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4959.124073] LustreError: 82436:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff98a2c3023800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff98a2c8b52f00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4959.148806] LustreError: 82436:0:(ldlm_resource.c:981:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4959.760181] LustreError: 82422:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 4966.760124] LustreError: lustre-MDT0000-mdc-ffff98a2c3023800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4967.228938] LustreError: 82478:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 4970.082356] LustreError: lustre-MDT0000-mdc-ffff98a2c3023800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4970.164779] LustreError: 82498:0:(file.c:6118:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4970.170198] LustreError: 82498:0:(file.c:6118:ll_inode_revalidate_fini()) Skipped 14 previous similar messages
[ 4971.251948] LustreError: 82504:0:(file.c:6118:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4971.289780] LustreError: 82504:0:(file.c:6118:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4971.320109] LustreError: 82478:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 4971.342686] LustreError: 82478:0:(file.c:249:ll_close_inode_openhandle()) lustre-clilmv-ffff98a2c3023800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 4973.747178] LustreError: 82526:0:(file.c:6118:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4973.763493] LustreError: 82526:0:(file.c:6118:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4977.334914] LustreError: 82553:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 4980.449542] LustreError: lustre-MDT0000-mdc-ffff98a2c3023800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4980.480770] LustreError: 82568:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff98a2c3023800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff98a2ca908900) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4980.511817] LustreError: 82568:0:(ldlm_resource.c:981:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4981.435562] LustreError: 82553:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 4988.648298] LustreError: lustre-MDT0000-mdc-ffff98a2c3023800: operation ldlm_enqueue to node 192.168.201.148@tcp failed: rc = -107
[ 4988.673578] LustreError: Skipped 5 previous similar messages
[ 4988.772527] LustreError: lustre-MDT0000-mdc-ffff98a2c3023800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4989.240305] LustreError: 82609:0:(ldlm_flock.c:856:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 4992.434143] LustreError: lustre-MDT0000-mdc-ffff98a2c3023800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4992.466428] LustreError: 82624:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff98a2c3023800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff98a2c8b52a00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4992.526338] LustreError: 82624:0:(ldlm_resource.c:981:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4992.556411] LustreError: 82629:0:(file.c:6118:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4992.584382] LustreError: 82629:0:(file.c:6118:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 4993.249438] LustreError: 82609:0:(ldlm_flock.c:856:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 4996.353284] LustreError: lustre-MDT0000-mdc-ffff98a2c3023800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4997.448714] LustreError: 82637:0:(file.c:249:ll_close_inode_openhandle()) lustre-clilmv-ffff98a2c3023800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5005.822396] LustreError: 82711:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5005.850922] LustreError: 82711:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5009.840246] LustreError: 82711:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5009.849932] LustreError: 82711:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5009.894350] LustreError: lustre-MDT0000-mdc-ffff98a2c3023800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5009.921594] LustreError: 82711:0:(file.c:249:ll_close_inode_openhandle()) lustre-clilmv-ffff98a2c3023800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -5
[ 5018.378364] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 16:05:24 (1764795924)
[ 5037.944748] LustreError: 67682:0:(ldlm_lockd.c:2915:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5042.016172] LustreError: 67682:0:(ldlm_lockd.c:2915:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5048.533852] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 16:05:54 (1764795954)
[ 5097.375524] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 16:06:43 (1764796003)
[ 5159.356162] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 16:07:45 (1764796065)
[ 5191.835640] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 16:08:17 (1764796097)
[ 5192.423373] Lustre: Mounted lustre-client
[ 5315.695232] LustreError: 86381:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff98a2cae08800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 5315.716454] LustreError: 86381:0:(lov_obd.c:783:lov_cleanup()) Skipped 1 previous similar message
[ 5315.734480] LustreError: 86381:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 5315.742189] LustreError: 86381:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 5315.772158] Lustre: Unmounted lustre-client
[ 5322.986355] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 16:10:28 (1764796228)
[ 5323.581924] LustreError: 86997:0:(osc_request.c:2978:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 4000ms
[ 5326.704843] LustreError: lustre-OST0000-osc-ffff98a2c3023800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5326.716115] LustreError: 87084:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-OST0000-osc-ffff98a2c3023800: namespace resource [0x240000400:0x7d0d:0x0].0x0 (ffff98a2c3183c00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5326.739125] LustreError: 87084:0:(ldlm_resource.c:981:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5327.680209] LustreError: 86997:0:(osc_request.c:2978:osc_build_rpc()) cfs_fail_timeout id 414 awake
[ 5327.694483] Lustre: 2364:0:(llite_lib.c:4237:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.148@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5327.723721] Lustre: lustre-OST0000-osc-ffff98a2c3023800: Connection restored to 192.168.201.148@tcp (at 192.168.201.148@tcp)
[ 5327.745244] Lustre: Skipped 14 previous similar messages
[ 5334.444897] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 16:10:40 (1764796240)
[ 5397.856629] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 16:11:44 (1764796304)
[ 5398.979130] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 5400.240747] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 16:11:46 (1764796306)
[ 5463.568228] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 5464.919066] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 16:12:51 (1764796371)
[ 5523.256733] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 16:13:48 (1764796428)
[ 5525.126697] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 5526.600681] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 16:13:52 (1764796432)
[ 5527.705459] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 5529.192682] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 16:13:55 (1764796435)
[ 5556.871438] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 16:14:22 (1764796462)
[ 5565.643134] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5587.872200] Lustre: 2362:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764796479/real 1764796479]  req@ffff98a2c64c9c00 x1850517512775296/t0(0) o400->MGC192.168.201.148@tcp@192.168.201.148@tcp:26/25 lens 224/224 e 0 to 1 dl 1764796495 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5587.886393] Lustre: 2362:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
[ 5587.893432] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 192.168.201.148@tcp) was lost; in progress operations using this service will fail
[ 5587.908044] LustreError: Skipped 2 previous similar messages
[ 5593.061174] Lustre: lustre-MDT0000-mdc-ffff98a2c3023800: Connection to lustre-MDT0000 (at 192.168.201.148@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5593.076913] Lustre: Skipped 16 previous similar messages
[ 5598.265730] Lustre: Evicted from MGS (at 192.168.201.148@tcp) after server handle changed from 0x50b7f38e58f29070 to 0x50b7f38e58f2aafd
[ 5598.275362] Lustre: Skipped 2 previous similar messages
[ 5601.103964] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5602.936305] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5612.134852] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 16:15:18 (1764796518)
[ 5614.195450] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 5615.805340] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 16:15:21 (1764796521)
[ 5628.913262] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 192.168.201.148@tcp) was lost; in progress operations using this service will fail
[ 5640.791798] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 16:15:46 (1764796546)
[ 5669.676065] Lustre: Evicted from MGS (at 192.168.201.148@tcp) after server handle changed from 0x50b7f38e58f2acfc to 0x50b7f38e58f2b24a
[ 5669.685076] Lustre: Skipped 1 previous similar message
[ 5678.269771] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 16:16:24 (1764796584)
[ 5684.375500] LustreError: lustre-OST0000-osc-ffff98a2c3023800: operation ost_setattr to node 192.168.201.148@tcp failed: rc = -19
[ 5684.383107] LustreError: Skipped 4 previous similar messages
[ 5725.956484] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5728.899388] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5813.231146] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 192.168.201.148@tcp) was lost; in progress operations using this service will fail
[ 5813.242306] LustreError: Skipped 1 previous similar message
[ 5813.253983] Lustre: Evicted from MGS (at 192.168.201.148@tcp) after server handle changed from 0x50b7f38e58f2b24a to 0x50b7f38e58f2bead
[ 5818.840053] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5820.115376] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5850.980788] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5852.598923] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5949.526654] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 16:20:55 (1764796855)
[ 5991.139913] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5993.718355] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6217.338549] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 16:25:23 (1764797123)
[ 6355.941277] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 192.168.201.148@tcp) was lost; in progress operations using this service will fail
[ 6355.952816] LustreError: Skipped 1 previous similar message
[ 6355.961072] Lustre: lustre-MDT0000-mdc-ffff98a2c3023800: Connection to lustre-MDT0000 (at 192.168.201.148@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6355.973671] Lustre: Evicted from MGS (at 192.168.201.148@tcp) after server handle changed from 0x50b7f38e58f2c2b2 to 0x50b7f38e59026497
[ 6355.982638] Lustre: Skipped 6 previous similar messages
[ 6355.986842] Lustre: Skipped 1 previous similar message
[ 6355.994815] Lustre: MGC192.168.201.148@tcp: Connection restored to 192.168.201.148@tcp (at 192.168.201.148@tcp)
[ 6355.998259] Lustre: Skipped 13 previous similar messages
[ 6384.095555] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 16:28:10 (1764797290)
[ 6385.526684] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 6387.149783] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 16:28:13 (1764797293)
[ 6389.346616] LustreError: lustre-MDT0000-mdc-ffff98a2c3023800: operation ldlm_enqueue to node 192.168.201.148@tcp failed: rc = -107
[ 6389.358878] LustreError: Skipped 525 previous similar messages
[ 6389.433458] LustreError: lustre-MDT0000-mdc-ffff98a2c3023800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6396.106550] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 16:28:22 (1764797302)
[ 6563.117083] LustreError: lustre-OST0000-osc-ffff98a2c3023800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6564.800063] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 16:31:10 (1764797470)
[ 6589.600166] Lustre: 2362:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764797476/real 1764797476]  req@ffff98a2c7d25880 x1850517530498432/t0(0) o4->lustre-OST0000-osc-ffff98a2c3023800@192.168.201.148@tcp:6/4 lens 4584/448 e 0 to 1 dl 1764797496 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 6589.641289] Lustre: 2362:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 17 previous similar messages
[ 6605.952733] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 16:31:52 (1764797512)
[ 6607.261837] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 6608.824967] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 16:31:54 (1764797514)
[ 6609.895432] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 6611.239178] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 16:31:57 (1764797517)
[ 6643.234674] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 16:32:29 (1764797549)
[ 6675.949954] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 192.168.201.148@tcp) was lost; in progress operations using this service will fail
[ 6675.970823] Lustre: Evicted from MGS (at 192.168.201.148@tcp) after server handle changed from 0x50b7f38e59026497 to 0x50b7f38e59028c4b
[ 6675.986056] Lustre: Skipped 1 previous similar message
[ 6687.690645] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 16:33:13 (1764797593)
[ 6689.032826] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 6690.531730] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 16:33:16 (1764797596)
[ 6691.878279] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 6693.130543] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 16:33:19 (1764797599)
[ 6699.194749] LustreError: 105858:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff98a2c3023800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 6699.207418] LustreError: 105858:0:(lov_obd.c:783:lov_cleanup()) Skipped 1 previous similar message
[ 6699.233039] LustreError: 105858:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup
[ 6699.235967] LustreError: 105858:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 6699.287963] Lustre: Unmounted lustre-client
[ 6700.258516] Lustre: Mounted lustre-client
[ 6702.338207] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6727.681707] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: total 1823