[ 1562.538780] LustreError: lustre-OST0000-osc-ffff9e20c2e8e000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1562.546168] Lustre: 2367:0:(llite_lib.c:4190:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.101@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1565.157839] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 20:28:13 (1777422493)
[ 1566.085606] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1566.777714] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 20:28:15 (1777422495)
[ 1567.424098] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1568.014438] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 20:28:16 (1777422496)
[ 1586.146585] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 192.168.203.101@tcp) was lost; in progress operations using this service will fail
[ 1586.166493] Lustre: Evicted from MGS (at 192.168.203.101@tcp) after server handle changed from 0x110b41c0714949c5 to 0x110b41c071496db3
[ 1677.331785] LustreError: lustre-MDT0000-mdc-ffff9e20c2e8e000: operation ldlm_enqueue to node 192.168.203.101@tcp failed: rc = -19
[ 1677.336146] LustreError: Skipped 2 previous similar messages
[ 1693.668320] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 192.168.203.101@tcp) was lost; in progress operations using this service will fail
[ 1693.698428] Lustre: Evicted from MGS (at 192.168.203.101@tcp) after server handle changed from 0x110b41c071496db3 to 0x110b41c0714e0e18
[ 1696.265136] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 20:30:24 (1777422624)
[ 1696.419740] Lustre: *** cfs_fail_loc=305, val=0***
[ 1696.421414] Lustre: Skipped 4 previous similar messages
[ 1729.508780] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 192.168.203.101@tcp) was lost; in progress operations using this service will fail
[ 1729.525477] Lustre: Evicted from MGS (at 192.168.203.101@tcp) after server handle changed from 0x110b41c0714e0e18 to 0x110b41c0714e1319
[ 1732.454836] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1732.980170] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1736.412541] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 20:31:04 (1777422664)
[ 1744.204284] LustreError: lustre-MDT0000-mdc-ffff9e20c2e8e000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 1744.865409] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 192.168.203.101@tcp) was lost; in progress operations using this service will fail
[ 1744.873098] Lustre: Evicted from MGS (at 192.168.203.101@tcp) after server handle changed from 0x110b41c0714e1319 to 0x110b41c0714e187c
[ 1750.780140] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 20:31:19 (1777422679)
[ 1755.886204] LustreError: lustre-OST0000-osc-ffff9e20c2e8e000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1763.550994] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 20:31:31 (1777422691)
[ 1790.947431] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 192.168.203.101@tcp) was lost; in progress operations using this service will fail
[ 1790.983398] Lustre: Evicted from MGS (at 192.168.203.101@tcp) after server handle changed from 0x110b41c0714e187c to 0x110b41c0714eb20c
[ 1793.442506] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1794.265844] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1856.714044] LustreError: lustre-MDT0000-mdc-ffff9e20c2e8e000: operation mds_reint to node 192.168.203.101@tcp failed: rc = -19
[ 1856.718548] LustreError: Skipped 3 previous similar messages
[ 1872.870632] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 192.168.203.101@tcp) was lost; in progress operations using this service will fail
[ 1872.908089] Lustre: Evicted from MGS (at 192.168.203.101@tcp) after server handle changed from 0x110b41c0714eb20c to 0x110b41c071522310
[ 1872.913732] Lustre: MGC192.168.203.101@tcp: Connection restored to 192.168.203.101@tcp (at 192.168.203.101@tcp)
[ 1872.917157] Lustre: Skipped 21 previous similar messages
[ 1876.347277] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1877.239485] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1939.972169] Lustre: lustre-MDT0000-mdc-ffff9e20c2e8e000: Connection to lustre-MDT0000 (at 192.168.203.101@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 1939.980745] Lustre: Skipped 17 previous similar messages
[ 1954.785865] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 192.168.203.101@tcp) was lost; in progress operations using this service will fail
[ 1954.794767] Lustre: Evicted from MGS (at 192.168.203.101@tcp) after server handle changed from 0x110b41c071522310 to 0x110b41c07155c1da
[ 1961.426138] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1962.351481] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1986.591111] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 20:35:14 (1777422914)
[ 2004.786823] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2021.346062] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 192.168.203.101@tcp) was lost; in progress operations using this service will fail
[ 2021.349301] LustreError: Skipped 1 previous similar message
[ 2021.351267] Lustre: 15817:0:(mgc_request.c:1911:mgc_process_log()) MGC192.168.203.101@tcp: IR log lustre-cliir failed, not fatal: rc = -5
[ 2021.353172] Lustre: Evicted from MGS (at 192.168.203.101@tcp) after server handle changed from 0x110b41c071573c74 to 0x110b41c071574710
[ 2021.357495] Lustre: Skipped 1 previous similar message
[ 2021.948266] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2043.412556] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2071.023446] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2107.873848] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2134.125932] LustreError: lustre-MDT0000-mdc-ffff9e20c2e8e000: operation ldlm_enqueue to node 192.168.203.101@tcp failed: rc = -19
[ 2134.129216] LustreError: Skipped 7 previous similar messages
[ 2149.632780] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2193.379402] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 192.168.203.101@tcp) was lost; in progress operations using this service will fail
[ 2193.383313] LustreError: Skipped 4 previous similar messages
[ 2193.391596] Lustre: Evicted from MGS (at 192.168.203.101@tcp) after server handle changed from 0x110b41c0715a4e11 to 0x110b41c0715bdeca
[ 2193.394420] Lustre: Skipped 4 previous similar messages
[ 2217.880711] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 20:39:06 (1777423146)
[ 2249.462829] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2250.735880] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 2559.305231] Lustre: lustre-OST0000-osc-ffff9e20c2e8e000: Connection to lustre-OST0000 (at 192.168.203.101@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2559.316420] Lustre: Skipped 8 previous similar messages
[ 2578.741387] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2579.626259] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 2891.225740] LustreError: lustre-OST0000-osc-ffff9e20c2e8e000: operation ldlm_enqueue to node 192.168.203.101@tcp failed: rc = -107
[ 2891.231696] LustreError: Skipped 4 previous similar messages
[ 2911.076484] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2911.768354] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3182.474485] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 20:55:10 (1777424110)
[ 3198.431198] Lustre: 50345:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777424111/real 1777424111]  req@ffff9e20d81a0700 x1863761080936832/t0(0) o101->lustre-MDT0000-mdc-ffff9e20c2e8e000@192.168.203.101@tcp:12/10 lens 576/1152 e 0 to 1 dl 1777424127 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3198.440596] Lustre: 50345:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
[ 3198.444073] Lustre: lustre-MDT0000-mdc-ffff9e20c2e8e000: Connection to lustre-MDT0000 (at 192.168.203.101@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3198.448464] Lustre: Skipped 1 previous similar message
[ 3198.460393] Lustre: lustre-MDT0000-mdc-ffff9e20c2e8e000: Connection restored to 192.168.203.101@tcp (at 192.168.203.101@tcp)
[ 3198.463672] Lustre: Skipped 18 previous similar messages
[ 3201.167393] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 20:55:29 (1777424129)
[ 3220.261613] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 20:55:48 (1777424148)
[ 3239.333112] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 20:56:07 (1777424167)
[ 3239.453717] Lustre: Mounted lustre-client
[ 3264.994057] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 192.168.203.101@tcp) was lost; in progress operations using this service will fail
[ 3264.999880] Lustre: Evicted from MGS (at 192.168.203.101@tcp) after server handle changed from 0x110b41c0715bdeca to 0x110b41c0718f1c8b
[ 3267.192969] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3267.726698] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3269.074795] LustreError: 53109:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff9e20c8980000: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 3269.080323] LustreError: 53109:0:(lov_obd.c:783:lov_cleanup()) Skipped 1 previous similar message
[ 3269.098617] Lustre: Unmounted lustre-client
[ 3270.942797] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 20:56:39 (1777424199)
[ 3276.255135] Lustre: 2368:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777424188/real 1777424188]  req@ffff9e20d81a0380 x1863761080956032/t0(0) o400->lustre-MDT0000-mdc-ffff9e20c2e8e000@192.168.203.101@tcp:12/10 lens 224/224 e 0 to 1 dl 1777424204 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3276.263072] Lustre: 2368:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 3306.980208] Lustre: lustre-OST0000-osc-ffff9e20c2e8e000: Connection restored to 192.168.203.101@tcp (at 192.168.203.101@tcp)
[ 3306.982842] Lustre: Skipped 5 previous similar messages
[ 3435.999119] Lustre: 2367:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777424348/real 1777424348]  req@ffff9e20c8ba4000 x1863761080984448/t0(0) o4->lustre-OST0000-osc-ffff9e20c2e8e000@192.168.203.101@tcp:6/4 lens 488/448 e 0 to 1 dl 1777424364 ref 2 fl Rpc:XQr/602/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 3436.009279] Lustre: 2367:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 88 previous similar messages
[ 3467.748305] Lustre: lustre-OST0000-osc-ffff9e20c2e8e000: Connection restored to 192.168.203.101@tcp (at 192.168.203.101@tcp)
[ 3467.752096] Lustre: Skipped 9 previous similar messages
[ 3552.939849] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 21:01:21 (1777424481)
[ 3595.720835] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 21:02:04 (1777424524)
[ 3597.013623] LustreError: 55271:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff9e20c2e8e000: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 3597.017599] LustreError: 55271:0:(lov_obd.c:783:lov_cleanup()) Skipped 1 previous similar message
[ 3597.042108] Lustre: Unmounted lustre-client
[ 3604.671899] Lustre: Mounted lustre-client
[ 3606.690706] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 21:02:15 (1777424535)
[ 3606.785567] LustreError: 56255:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 3607.843341] Lustre: *** cfs_fail_loc=305, val=0***
[ 3626.863122] LustreError: 56255:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout id 801 awake
[ 3629.188939] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 21:02:37 (1777424557)
[ 3629.328921] Lustre: Mounted lustre-client
[ 3629.741636] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 3639.990200] Lustre: Unmounted lustre-client
[ 3642.287391] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 21:02:50 (1777424570)
[ 3663.282855] LustreError: lustre-MDT0000-mdc-ffff9e20d0055800: operation mds_reint to node 192.168.203.101@tcp failed: rc = -19
[ 3680.737140] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 192.168.203.101@tcp) was lost; in progress operations using this service will fail
[ 3680.743238] Lustre: Evicted from MGS (at 192.168.203.101@tcp) after server handle changed from 0x110b41c0718f369a to 0x110b41c07192445c
[ 3720.890568] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 21:04:09 (1777424649)
[ 3722.698620] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 3726.847208] LustreError: lustre-MDT0000-mdc-ffff9e20d0055800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 3733.593079] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 21:04:22 (1777424662)
[ 3733.709810] Lustre: Mounted lustre-client
[ 3739.108296] LustreError: lustre-OST0000-osc-ffff9e20d0055800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 3749.343169] Lustre: 2367:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777424662/real 1777424662]  req@ffff9e20c48c2300 x1863761085545472/t0(0) o103->lustre-OST0000-osc-ffff9e20c8ae5800@192.168.203.101@tcp:17/18 lens 328/224 e 0 to 1 dl 1777424678 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295
[ 3749.353305] Lustre: 2367:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 59 previous similar messages
[ 3749.676941] LustreError: 59973:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff9e20c8ae5800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 3749.681992] LustreError: 59973:0:(lov_obd.c:783:lov_cleanup()) Skipped 1 previous similar message
[ 3749.703226] Lustre: Unmounted lustre-client
[ 3751.815705] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 21:04:40 (1777424680)
[ 3756.664947] LustreError: lustre-MDT0000-mdc-ffff9e20d0055800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 3756.668530] LustreError: 60613:0:(file.c:6460:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 3758.977422] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 21:04:47 (1777424687)
[ 3759.036130] LustreError: 61311:0:(recover.c:330:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 3762.363801] LustreError: lustre-MDT0000-mdc-ffff9e20d0055800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 3762.368886] LustreError: 61332:0:(import.c:291:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 3762.371485] LustreError: 61311:0:(recover.c:330:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=1669
[ 3762.374170] LustreError: 61311:0:(import.c:716:ptlrpc_connect_import_locked()) already connecting
[ 3762.386556] LustreError: 61337:0:(file.c:6460:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3763.413477] LustreError: 61343:0:(file.c:6460:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3763.417718] LustreError: 61343:0:(file.c:6460:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 3765.502048] LustreError: 61366:0:(file.c:6460:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3765.505053] LustreError: 61366:0:(file.c:6460:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 3769.653429] LustreError: 61410:0:(file.c:6460:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3769.657097] LustreError: 61410:0:(file.c:6460:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 3772.385742] Lustre: lustre-MDT0000-mdc-ffff9e20d0055800: Connection restored to 192.168.203.101@tcp (at 192.168.203.101@tcp)
[ 3772.389040] Lustre: Skipped 15 previous similar messages
[ 3775.045861] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 21:05:03 (1777424703)
[ 3795.565036] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3796.084275] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3799.868748] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 21:05:28 (1777424728)
[ 3803.619333] Lustre: lustre-OST0000-osc-ffff9e20d0055800: Connection to lustre-OST0000 (at 192.168.203.101@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3803.623340] Lustre: Skipped 32 previous similar messages
[ 3818.202268] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3818.704173] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3822.838146] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 21:05:51 (1777424751)
[ 3866.514377] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3867.030133] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3870.478687] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 21:06:38 (1777424798)
[ 3889.248866] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3889.829293] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3892.081173] LustreError: 66978:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff9e20d0055800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 3892.084783] LustreError: 66978:0:(lov_obd.c:783:lov_cleanup()) Skipped 1 previous similar message
[ 3892.106143] Lustre: Unmounted lustre-client
[ 3915.193928] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3916.670196] Lustre: Mounted lustre-client
[ 3918.960278] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 21:07:27 (1777424847)
[ 3919.751045] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 3920.282761] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 21:07:28 (1777424848)
[ 3930.453168] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 21:07:38 (1777424858)
[ 3930.933466] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 3931.490093] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 21:07:39 (1777424859)
[ 3931.921198] Lustre: *** cfs_fail_loc=805, val=0***
[ 3933.615165] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 3947.489717] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 192.168.203.101@tcp) was lost; in progress operations using this service will fail
[ 3947.495678] LustreError: Skipped 1 previous similar message
[ 3947.501949] Lustre: Evicted from MGS (at 192.168.203.101@tcp) after server handle changed from 0x110b41c071967a40 to 0x110b41c071967d50
[ 3947.505345] Lustre: Skipped 1 previous similar message
[ 3949.358154] LustreError: 70308:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff9e20d903e800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 3949.361402] LustreError: 70308:0:(lov_obd.c:783:lov_cleanup()) Skipped 1 previous similar message
[ 3949.378124] Lustre: Unmounted lustre-client
[ 3951.817338] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 21:08:00 (1777424880)
[ 3967.986271] LustreError: 2364:0:(client.c:3418:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff9e20c607b800 x1863761085607680/t94489280516(94489280516) o101->lustre-MDT0000-mdc-ffff9e20c2e88000@192.168.203.101@tcp:12/10 lens 664/608 e 0 to 0 dl 1777424912 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 3970.343118] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3970.902869] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3974.285033] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 21:08:22 (1777424902)
[ 3974.829710] LustreError: lustre-OST0000-osc-ffff9e20c2e88000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 3974.835721] Lustre: 2367:0:(llite_lib.c:4190:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.101@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 3974.856503] LustreError: 72310:0:(ldlm_resource.c:1170:ldlm_resource_complain()) lustre-OST0000-osc-ffff9e20c2e88000: namespace resource [0x240000400:0x8122:0x0].0x0 (ffff9e20c5fd7700) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 3978.259722] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 21:08:26 (1777424906)
[ 3978.787055] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 3979.409964] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 21:08:27 (1777424907)
[ 3979.969180] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 3980.529468] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 21:08:28 (1777424908)
[ 3981.073338] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 3981.658962] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 21:08:30 (1777424910)
[ 3982.197705] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 3982.772583] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 21:08:31 (1777424911)
[ 3983.247810] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 3983.777018] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 21:08:32 (1777424912)
[ 3984.284880] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 3984.881162] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 21:08:33 (1777424913)
[ 3985.405307] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 3985.942938] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 21:08:34 (1777424914)
[ 3986.473681] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 3987.030199] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 21:08:35 (1777424915)
[ 3987.516119] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 3988.061158] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 21:08:36 (1777424916)
[ 3988.577091] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 3989.141792] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 21:08:37 (1777424917)
[ 3989.651506] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 3990.227348] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 21:08:38 (1777424918)
[ 3991.077390] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 3991.654336] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 21:08:40 (1777424920)
[ 4000.684506] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 21:08:49 (1777424929)
[ 4023.681903] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 21:09:12 (1777424952)
[ 4023.802403] Lustre: *** cfs_fail_loc=51b, val=3***
[ 4029.824970] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 21:09:18 (1777424958)
[ 4029.899507] Lustre: *** cfs_fail_loc=51b, val=4***
[ 4036.101845] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 21:09:24 (1777424964)
[ 4036.185569] Lustre: *** cfs_fail_loc=50f, val=3***
[ 4039.574783] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 21:09:28 (1777424968)
[ 4039.636326] Lustre: *** cfs_fail_loc=50f, val=4***
[ 4042.942619] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 21:09:31 (1777424971)
[ 4043.022434] Lustre: *** cfs_fail_loc=510, val=3***
[ 4046.207476] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 21:09:34 (1777424974)
[ 4046.285937] Lustre: *** cfs_fail_loc=51b, val=3***
[ 4052.137400] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 21:09:40 (1777424980)
[ 4113.577812] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 21:10:42 (1777425042)
[ 4113.666321] LustreError: 82487:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 4115.954740] LustreError: lustre-MDT0000-mdc-ffff9e20c2e88000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4115.960204] LustreError: 82502:0:(file.c:249:ll_close_inode_openhandle()) lustre-clilmv-ffff9e20c2e88000: inode [0x20000a042:0x8:0x0] mdc close failed: rc = -108
[ 4115.971506] LustreError: 82502:0:(ldlm_resource.c:1170:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9e20c2e88000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9e20c618ee00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4115.975204] LustreError: 82507:0:(file.c:6460:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4115.979012] LustreError: 82502:0:(ldlm_resource.c:1170:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4115.982321] LustreError: 82507:0:(file.c:6460:ll_inode_revalidate_fini()) Skipped 20 previous similar messages
[ 4117.727113] LustreError: 82487:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 4119.754982] LustreError: 82528:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 4122.019542] LustreError: lustre-MDT0000-mdc-ffff9e20c2e88000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4122.025767] LustreError: 82543:0:(ldlm_resource.c:1170:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9e20c2e88000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9e20c898e600) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4122.033929] LustreError: 82543:0:(ldlm_resource.c:1170:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4123.815158] LustreError: 82528:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 4130.045745] LustreError: lustre-MDT0000-mdc-ffff9e20c2e88000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4130.153177] LustreError: 82584:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 4132.415614] LustreError: lustre-MDT0000-mdc-ffff9e20c2e88000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4132.435454] LustreError: 82603:0:(file.c:6460:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4132.438040] LustreError: 82603:0:(file.c:6460:ll_inode_revalidate_fini()) Skipped 2 previous similar messages
[ 4134.215137] LustreError: 82584:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 4134.219056] LustreError: 82584:0:(file.c:249:ll_close_inode_openhandle()) lustre-clilmv-ffff9e20c2e88000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 4134.223631] LustreError: 82584:0:(file.c:249:ll_close_inode_openhandle()) Skipped 4 previous similar messages
[ 4138.704835] LustreError: 82659:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 4140.986523] LustreError: lustre-MDT0000-mdc-ffff9e20c2e88000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4140.992537] LustreError: 82674:0:(ldlm_resource.c:1170:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9e20c2e88000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9e20c618e700) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4140.997423] LustreError: 82674:0:(ldlm_resource.c:1170:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4142.767103] LustreError: 82659:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 4149.046172] LustreError: lustre-MDT0000-mdc-ffff9e20c2e88000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4149.168041] LustreError: 82715:0:(ldlm_flock.c:856:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 4151.470775] LustreError: lustre-MDT0000-mdc-ffff9e20c2e88000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4151.477809] LustreError: 82730:0:(ldlm_resource.c:1170:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9e20c2e88000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9e20c898e900) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4151.484567] LustreError: 82730:0:(ldlm_resource.c:1170:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4153.231095] LustreError: 82715:0:(ldlm_flock.c:856:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 4155.556043] LustreError: lustre-MDT0000-mdc-ffff9e20c2e88000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4157.319583] LustreError: 82743:0:(file.c:249:ll_close_inode_openhandle()) lustre-clilmv-ffff9e20c2e88000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 4166.380069] LustreError: lustre-MDT0000-mdc-ffff9e20c2e88000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4166.386535] LustreError: 82832:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 4166.389890] LustreError: 82832:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) Skipped 2 previous similar messages
[ 4166.405034] LustreError: 82837:0:(file.c:6460:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4166.409908] LustreError: 82837:0:(file.c:6460:ll_inode_revalidate_fini()) Skipped 53 previous similar messages
[ 4167.935562] LustreError: 82815:0:(file.c:249:ll_close_inode_openhandle()) lustre-clilmv-ffff9e20c2e88000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 4170.447120] LustreError: 82832:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 4170.449907] LustreError: 82832:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) Skipped 2 previous similar messages
[ 4172.568123] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 21:11:41 (1777425101)
[ 4194.641083] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 21:12:03 (1777425123)
[ 4239.819349] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 21:12:48 (1777425168)
[ 4299.010050] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 21:13:47 (1777425227)
[ 4324.639291] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 21:14:13 (1777425253)
[ 4324.751721] Lustre: Mounted lustre-client
[ 4324.752634] Lustre: Skipped 1 previous similar message
[ 4445.446722] LustreError: 86526:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff9e20c5653800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 4445.451284] LustreError: 86526:0:(lov_obd.c:783:lov_cleanup()) Skipped 1 previous similar message
[ 4445.469131] Lustre: Unmounted lustre-client
[ 4447.575933] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 21:16:16 (1777425376)
[ 4447.753975] LustreError: 87141:0:(osc_request.c:2968:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 4000ms
[ 4447.757788] LustreError: 87141:0:(osc_request.c:2968:osc_build_rpc()) Skipped 1 previous similar message
[ 4449.524469] LustreError: lustre-OST0000-osc-ffff9e20c2e88000: operation ost_statfs to node 192.168.203.101@tcp failed: rc = -107
[ 4449.527434] LustreError: Skipped 12 previous similar messages
[ 4449.529554] Lustre: lustre-OST0000-osc-ffff9e20c2e88000: Connection to lustre-OST0000 (at 192.168.203.101@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4449.534336] Lustre: Skipped 21 previous similar messages
[ 4449.538619] LustreError: lustre-OST0000-osc-ffff9e20c2e88000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4449.544681] LustreError: 87228:0:(ldlm_resource.c:1170:ldlm_resource_complain()) lustre-OST0000-osc-ffff9e20c2e88000: namespace resource [0x240000400:0x814d:0x0].0x0 (ffff9e20c618e000) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4449.551283] LustreError: 87228:0:(ldlm_resource.c:1170:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4451.815103] LustreError: 87141:0:(osc_request.c:2968:osc_build_rpc()) cfs_fail_timeout id 414 awake
[ 4451.817312] LustreError: 87141:0:(osc_request.c:2968:osc_build_rpc()) Skipped 1 previous similar message
[ 4451.819424] Lustre: 2365:0:(llite_lib.c:4190:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.101@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 4451.824880] Lustre: lustre-OST0000-osc-ffff9e20c2e88000: Connection restored to 192.168.203.101@tcp (at 192.168.203.101@tcp)
[ 4451.828527] Lustre: Skipped 24 previous similar messages
[ 4453.706285] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 21:16:22 (1777425382)
[ 4511.711190] Lustre: 87806:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777425384/real 1777425384]  req@ffff9e20c48c1c00 x1863761085766144/t0(0) o101->lustre-MDT0000-mdc-ffff9e20c2e88000@192.168.203.101@tcp:12/10 lens 328/344 e 0 to 1 dl 1777425440 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 4511.720529] Lustre: 87806:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 10 previous similar messages
[ 4514.027381] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 21:17:22 (1777425442)
[ 4514.525261] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 4515.135793] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 21:17:23 (1777425443)
[ 4574.328439] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 4574.892497] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 21:18:23 (1777425503)
[ 4632.037300] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 21:19:20 (1777425560)
[ 4632.796560] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 4633.349625] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 21:19:21 (1777425561)
[ 4633.847883] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 4634.400278] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 21:19:22 (1777425562)
[ 4643.872022] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 21:19:32 (1777425572)
[ 4646.967235] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4662.242243] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 192.168.203.101@tcp) was lost; in progress operations using this service will fail
[ 4662.247112] LustreError: Skipped 2 previous similar messages
[ 4662.251785] Lustre: Evicted from MGS (at 192.168.203.101@tcp) after server handle changed from 0x110b41c071968855 to 0x110b41c07196a256
[ 4662.255582] Lustre: Skipped 2 previous similar messages
[ 4665.724844] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4666.297887] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4670.297942] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 21:19:58 (1777425598)
[ 4671.133139] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 4671.680161] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 21:20:00 (1777425600)
[ 4679.230574] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 21:20:07 (1777425607)
[ 4691.300382] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 21:20:19 (1777425619)
[ 4714.197081] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4715.229642] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4794.852745] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4795.318894] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4814.199105] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4814.727449] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4877.999551] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 21:23:26 (1777425806)
[ 4902.577230] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4903.691870] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5117.505470] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 21:27:25 (1777426045)
[ 5148.643073] Lustre: lustre-MDT0000-mdc-ffff9e20c2e88000: Connection to lustre-MDT0000 (at 192.168.203.101@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5148.643298] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 192.168.203.101@tcp) was lost; in progress operations using this service will fail
[ 5148.648276] Lustre: Skipped 10 previous similar messages
[ 5148.652611] LustreError: Skipped 4 previous similar messages
[ 5148.774274] Lustre: Evicted from MGS (at 192.168.203.101@tcp) after server handle changed from 0x110b41c07196ba7b to 0x110b41c071a65d6a
[ 5148.777679] Lustre: Skipped 4 previous similar messages
[ 5148.780669] Lustre: MGC192.168.203.101@tcp: Connection restored to 192.168.203.101@tcp (at 192.168.203.101@tcp)
[ 5148.783850] Lustre: Skipped 13 previous similar messages
[ 5169.325948] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 21:28:17 (1777426097)
[ 5169.773771] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 5170.277183] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 21:28:18 (1777426098)
[ 5170.859943] LustreError: lustre-MDT0000-mdc-ffff9e20c2e88000: operation ldlm_enqueue to node 192.168.203.101@tcp failed: rc = -107
[ 5170.863222] LustreError: Skipped 16 previous similar messages
[ 5170.866971] LustreError: lustre-MDT0000-mdc-ffff9e20c2e88000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5173.185621] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 21:28:21 (1777426101)
[ 5339.042033] LustreError: lustre-OST0000-osc-ffff9e20c2e88000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5339.554354] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 21:31:08 (1777426268)
[ 5361.631156] Lustre: 2367:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777426270/real 1777426270]  req@ffff9e20c6e40380 x1863761103435264/t0(0) o4->lustre-OST0000-osc-ffff9e20c2e88000@192.168.203.101@tcp:6/4 lens 4584/448 e 0 to 1 dl 1777426290 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 5361.640851] Lustre: 2367:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
[ 5372.914172] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 21:31:41 (1777426301)
[ 5373.411789] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 5373.964724] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 21:31:42 (1777426302)
[ 5374.428679] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 5374.973725] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 21:31:43 (1777426303)
[ 5399.064878] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 21:32:07 (1777426327)
[ 5431.101798] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 21:32:39 (1777426359)
[ 5431.606221] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 5432.140959] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 21:32:40 (1777426360)
[ 5432.659532] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 5433.217994] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 21:32:41 (1777426361)
[ 5437.812442] LustreError: 105979:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff9e20c2e88000: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 5437.817935] LustreError: 105979:0:(lov_obd.c:783:lov_cleanup()) Skipped 1 previous similar message
[ 5437.858752] Lustre: Unmounted lustre-client
[ 5438.274043] Lustre: Mounted lustre-client
[ 5439.221242] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5455.327106] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: total 1811