[ 1931.799270] LustreError: lustre-OST0000-osc-ffff90c820368000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1931.815990] Lustre: 2356:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.116@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1933.715877] Lustre: DEBUG MARKER: recovery-small test_24b: @@@@@@ IGNORE (bz5494): multiop didn't fail fsync: 0 or close: 5
[ 1939.904851] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 07:59:06 (1786017546)
[ 1941.745822] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1943.130963] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 07:59:09 (1786017549)
[ 1945.027742] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1946.341492] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 07:59:12 (1786017552)
[ 1949.089964] LustreError: lustre-MDT0000-mdc-ffff90c820368000: operation mds_reint to node 192.168.201.116@tcp failed: rc = -19
[ 1967.596831] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 192.168.201.116@tcp) was lost; in progress operations using this service will fail
[ 1967.623692] Lustre: Evicted from MGS (at 192.168.201.116@tcp) after server handle changed from 0xfc433508c53f1d60 to 0xfc433508c53f3833
[ 2069.949114] LustreError: lustre-MDT0000-mdc-ffff90c820368000: operation mds_reint to node 192.168.201.116@tcp failed: rc = -19
[ 2090.497217] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 192.168.201.116@tcp) was lost; in progress operations using this service will fail
[ 2090.554953] Lustre: Evicted from MGS (at 192.168.201.116@tcp) after server handle changed from 0xfc433508c53f3833 to 0xfc433508c5418019
[ 2103.122421] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 08:01:49 (1786017709)
[ 2103.664658] Lustre: *** cfs_fail_loc=305, val=0***
[ 2103.672913] Lustre: Skipped 2 previous similar messages
[ 2146.722127] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 192.168.201.116@tcp) was lost; in progress operations using this service will fail
[ 2146.756582] Lustre: Evicted from MGS (at 192.168.201.116@tcp) after server handle changed from 0xfc433508c5418019 to 0xfc433508c541a37b
[ 2148.212284] Lustre: 15933:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.116@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2173.995295] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2176.274491] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2186.692259] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 08:03:12 (1786017792)
[ 2203.118122] Lustre: lustre-MDT0000-mdc-ffff90c820368000: Connection to lustre-MDT0000 (at 192.168.201.116@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2203.139453] Lustre: Skipped 12 previous similar messages
[ 2203.146338] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 192.168.201.116@tcp) was lost; in progress operations using this service will fail
[ 2203.189796] Lustre: Evicted from MGS (at 192.168.201.116@tcp) after server handle changed from 0xfc433508c541a37b to 0xfc433508c541a9fd
[ 2203.203912] Lustre: MGC192.168.201.116@tcp: Connection restored to 192.168.201.116@tcp (at 192.168.201.116@tcp)
[ 2203.212626] Lustre: Skipped 15 previous similar messages
[ 2203.229594] LustreError: lustre-MDT0000-mdc-ffff90c820368000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2221.733794] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 08:03:47 (1786017827)
[ 2238.951822] LustreError: lustre-OST0000-osc-ffff90c820368000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2255.882699] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 08:04:21 (1786017861)
[ 2269.927000] LustreError: lustre-MDT0000-mdc-ffff90c820368000: operation mds_reint to node 192.168.201.116@tcp failed: rc = -19
[ 2269.940503] LustreError: Skipped 1 previous similar message
[ 2290.162780] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 192.168.201.116@tcp) was lost; in progress operations using this service will fail
[ 2290.177960] Lustre: Evicted from MGS (at 192.168.201.116@tcp) after server handle changed from 0xfc433508c541a9fd to 0xfc433508c541fa84
[ 2291.960385] Lustre: 15933:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.116@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2318.554095] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2320.835368] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2402.784248] Lustre: 2356:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786017994/real 1786017994]  req@ffff90c8184aad80 x1872773190197376/t0(0) o400->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 224/224 e 0 to 1 dl 1786018010 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2402.812156] Lustre: 2356:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 12 previous similar messages
[ 2402.817222] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 192.168.201.116@tcp) was lost; in progress operations using this service will fail
[ 2413.041631] Lustre: Evicted from MGS (at 192.168.201.116@tcp) after server handle changed from 0xfc433508c541fa84 to 0xfc433508c5438738
[ 2430.797912] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2432.736117] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2498.364921] LustreError: lustre-MDT0000-mdc-ffff90c820368000: operation ldlm_enqueue to node 192.168.201.116@tcp failed: rc = -19
[ 2498.382889] LustreError: Skipped 1 previous similar message
[ 2515.424280] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 192.168.201.116@tcp) was lost; in progress operations using this service will fail
[ 2525.820868] Lustre: Evicted from MGS (at 192.168.201.116@tcp) after server handle changed from 0xfc433508c5438738 to 0xfc433508c54542df
[ 2537.056220] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2539.291834] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2568.533965] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 08:09:34 (1786018174)
[ 2592.800295] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 192.168.201.116@tcp) was lost; in progress operations using this service will fail
[ 2592.846331] Lustre: Evicted from MGS (at 192.168.201.116@tcp) after server handle changed from 0xfc433508c54542df to 0xfc433508c5461ad5
[ 2594.607752] Lustre: 15933:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.116@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2610.920624] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2635.056906] Lustre: 15933:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.116@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2650.428743] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2679.225638] Lustre: 15933:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.116@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2695.363348] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2727.921104] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 192.168.201.116@tcp) was lost; in progress operations using this service will fail
[ 2727.929146] LustreError: Skipped 2 previous similar messages
[ 2727.955610] Lustre: Evicted from MGS (at 192.168.201.116@tcp) after server handle changed from 0xfc433508c54647f3 to 0xfc433508c546860c
[ 2727.961134] Lustre: Skipped 2 previous similar messages
[ 2729.834885] Lustre: 15933:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.116@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2746.922431] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2770.226261] LustreError: lustre-MDT0000-mdc-ffff90c820368000: operation ldlm_enqueue to node 192.168.201.116@tcp failed: rc = -19
[ 2770.239751] LustreError: Skipped 7 previous similar messages
[ 2798.212706] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2826.417965] Lustre: lustre-MDT0000-mdc-ffff90c820368000: Connection to lustre-MDT0000 (at 192.168.201.116@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2826.433498] Lustre: Skipped 9 previous similar messages
[ 2845.699977] Lustre: MGC192.168.201.116@tcp: Connection restored to 192.168.201.116@tcp (at 192.168.201.116@tcp)
[ 2845.718193] Lustre: Skipped 18 previous similar messages
[ 2855.623407] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2952.711448] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 08:15:58 (1786018558)
[ 3007.062429] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3009.094719] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3297.849701] LustreError: lustre-OST0000-osc-ffff90c820368000: operation ldlm_enqueue to node 192.168.201.116@tcp failed: rc = -19
[ 3297.870620] LustreError: Skipped 4 previous similar messages
[ 3340.109385] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3342.523939] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3629.530930] Lustre: lustre-OST0000-osc-ffff90c820368000: Connection to lustre-OST0000 (at 192.168.201.116@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3629.573729] Lustre: Skipped 3 previous similar messages
[ 3654.154302] Lustre: lustre-OST0000-osc-ffff90c820368000: Connection restored to 192.168.201.116@tcp (at 192.168.201.116@tcp)
[ 3654.160690] Lustre: Skipped 5 previous similar messages
[ 3673.268817] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3675.381767] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3926.898631] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 08:32:12 (1786019532)
[ 3944.416176] Lustre: 50704:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786019535/real 1786019535]  req@ffff90c820014700 x1872773212100992/t0(0) o101->lustre-MDT0000-mdc-ffff90c820368000@192.168.201.116@tcp:12/10 lens 576/1152 e 0 to 1 dl 1786019551 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3944.470201] Lustre: 50704:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
[ 3954.495051] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 08:32:40 (1786019560)
[ 3984.230969] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 08:33:10 (1786019590)
[ 4011.433590] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 08:33:37 (1786019617)
[ 4012.241110] Lustre: Mounted lustre-client
[ 4044.256596] Lustre: 2357:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786019635/real 1786019635]  req@ffff90c820158000 x1872773212120832/t0(0) o400->lustre-MDT0000-mdc-ffff90c820368000@192.168.201.116@tcp:12/10 lens 224/224 e 0 to 1 dl 1786019651 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4044.286271] Lustre: 2357:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 4044.296591] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 192.168.201.116@tcp) was lost; in progress operations using this service will fail
[ 4044.314455] LustreError: Skipped 3 previous similar messages
[ 4054.579899] Lustre: Evicted from MGS (at 192.168.201.116@tcp) after server handle changed from 0xfc433508c548f949 to 0xfc433508c55d0cbc
[ 4054.595066] Lustre: Skipped 3 previous similar messages
[ 4071.191361] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4073.276366] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4075.466934] Lustre: Unmounted lustre-client
[ 4083.546583] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 08:34:49 (1786019689)
[ 4206.560344] Lustre: 2356:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786019797/real 1786019797]  req@ffff90c820016680 x1872773212155776/t0(0) o4->lustre-OST0000-osc-ffff90c820368000@192.168.201.116@tcp:6/4 lens 488/448 e 0 to 1 dl 1786019813 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4206.583116] Lustre: 2356:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 50 previous similar messages
[ 4238.304337] Lustre: lustre-OST0000-osc-ffff90c820368000: Connection to lustre-OST0000 (at 192.168.201.116@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4238.318179] Lustre: Skipped 13 previous similar messages
[ 4254.510485] Lustre: lustre-OST0000-osc-ffff90c820368000: Connection restored to 192.168.201.116@tcp (at 192.168.201.116@tcp)
[ 4254.524398] Lustre: Skipped 15 previous similar messages
[ 4380.695989] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 08:39:46 (1786019986)
[ 4429.704415] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 08:40:35 (1786020035)
[ 4433.143906] Lustre: Unmounted lustre-client
[ 4454.344363] Lustre: Mounted lustre-client
[ 4465.280492] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 08:41:10 (1786020070)
[ 4466.070153] LustreError: 56625:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4467.137929] LustreError: 56625:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4467.359303] Lustre: *** cfs_fail_loc=305, val=0***
[ 4492.309622] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 08:41:37 (1786020097)
[ 4492.939538] Lustre: Mounted lustre-client
[ 4495.631584] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4505.942238] Lustre: Unmounted lustre-client
[ 4512.949386] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 08:41:58 (1786020118)
[ 4611.875450] LustreError: lustre-MDT0000-mdc-ffff90c818284000: operation mds_reint to node 192.168.201.116@tcp failed: rc = -19
[ 4611.881653] LustreError: Skipped 1 previous similar message
[ 4631.552517] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 192.168.201.116@tcp) was lost; in progress operations using this service will fail
[ 4631.576809] Lustre: Evicted from MGS (at 192.168.201.116@tcp) after server handle changed from 0xfc433508c55d26e7 to 0xfc433508c55fdab3
[ 4809.883162] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 08:46:55 (1786020415)
[ 4816.095276] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4831.213817] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 192.168.201.116@tcp) was lost; in progress operations using this service will fail
[ 4831.240183] Lustre: Evicted from MGS (at 192.168.201.116@tcp) after server handle changed from 0xfc433508c55fdab3 to 0xfc433508c564a622
[ 4831.394220] LustreError: lustre-MDT0000-mdc-ffff90c818284000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4850.604260] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 08:47:36 (1786020456)
[ 4851.380049] Lustre: Mounted lustre-client
[ 4859.054059] LustreError: lustre-OST0000-osc-ffff90c818284000: operation ldlm_enqueue to node 192.168.201.116@tcp failed: rc = -107
[ 4859.075993] Lustre: lustre-OST0000-osc-ffff90c818284000: Connection to lustre-OST0000 (at 192.168.201.116@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4859.117720] Lustre: Skipped 12 previous similar messages
[ 4859.144575] LustreError: lustre-OST0000-osc-ffff90c818284000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4869.088261] Lustre: 2357:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786020460/real 1786020460]  req@ffff90c818213800 x1872773216815488/t0(0) o103->lustre-OST0000-osc-ffff90c8191f1800@192.168.201.116@tcp:17/18 lens 328/224 e 0 to 1 dl 1786020476 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295
[ 4869.128697] Lustre: 2357:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 73 previous similar messages
[ 4869.169664] Lustre: lustre-OST0000-osc-ffff90c8191f1800: Connection restored to 192.168.201.116@tcp (at 192.168.201.116@tcp)
[ 4869.192522] Lustre: Skipped 13 previous similar messages
[ 4871.414839] Lustre: Unmounted lustre-client
[ 4878.422781] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 08:48:04 (1786020484)
[ 4886.267325] LustreError: lustre-MDT0000-mdc-ffff90c818284000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4886.296462] LustreError: 61005:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4894.461362] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 08:48:20 (1786020500)
[ 4894.770476] LustreError: 61709:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4899.809991] LustreError: 61709:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=0
[ 4899.836878] LustreError: 61709:0:(import.c:711:ptlrpc_connect_import_locked()) already connected
[ 4902.037275] LustreError: lustre-MDT0000-mdc-ffff90c818284000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4902.048208] LustreError: 61726:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4902.123724] LustreError: 61731:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4903.227076] LustreError: 61737:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4903.258851] LustreError: 61737:0:(file.c:6154:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4905.570804] LustreError: 61760:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4905.580641] LustreError: 61760:0:(file.c:6154:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4910.356161] LustreError: 61805:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4910.374560] LustreError: 61805:0:(file.c:6154:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4921.910917] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 08:48:47 (1786020527)
[ 4962.814850] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4964.522677] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4980.961578] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 08:49:45 (1786020585)
[ 5025.340279] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5027.282950] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5038.095385] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 08:50:44 (1786020644)
[ 5109.432417] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5110.845570] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5119.224572] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 08:52:05 (1786020725)
[ 5161.477306] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5163.434179] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5170.114125] Lustre: Unmounted lustre-client
[ 5232.880293] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5235.572556] Lustre: Mounted lustre-client
[ 5246.476692] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 08:54:11 (1786020851)
[ 5250.137338] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5253.332232] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 08:54:18 (1786020858)
[ 5294.836576] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 08:55:00 (1786020900)
[ 5296.643882] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5299.298933] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 08:55:04 (1786020904)
[ 5300.635916] Lustre: *** cfs_fail_loc=805, val=0***
[ 5300.678321] Lustre: Mounted lustre-client
[ 5307.737354] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5327.328506] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 192.168.201.116@tcp) was lost; in progress operations using this service will fail
[ 5337.605970] Lustre: Evicted from MGS (at 192.168.201.116@tcp) after server handle changed from 0xfc433508c564b047 to 0xfc433508c564b3f1
[ 5343.441451] Lustre: Unmounted lustre-client
[ 5353.401759] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 08:55:59 (1786020959)
[ 5395.999827] LustreError: 2355:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff90c8048d7480 x1872773216900096/t94489280516(94489280516) o101->lustre-MDT0000-mdc-ffff90c8204dc800@192.168.201.116@tcp:12/10 lens 648/608 e 0 to 0 dl 1786021019 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5407.550888] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5409.487421] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5419.902825] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 08:57:05 (1786021025)
[ 5421.770836] LustreError: lustre-OST0000-osc-ffff90c8204dc800: operation ost_write to node 192.168.201.116@tcp failed: rc = -107
[ 5421.788633] LustreError: Skipped 2 previous similar messages
[ 5421.812680] LustreError: lustre-OST0000-osc-ffff90c8204dc800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5421.832092] Lustre: 2357:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.116@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5421.901473] LustreError: 72848:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff90c8204dc800: namespace resource [0x240000400:0x3ee2:0x0].0x0 (ffff90c805d7d300) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5433.205821] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 08:57:18 (1786021038)
[ 5435.954989] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5438.286661] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 08:57:23 (1786021043)
[ 5439.572466] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5442.237786] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 08:57:27 (1786021047)
[ 5443.274365] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5444.817169] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 08:57:30 (1786021050)
[ 5447.066844] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5448.771200] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 08:57:34 (1786021054)
[ 5450.699849] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5452.368778] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 08:57:38 (1786021058)
[ 5454.124243] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5456.086892] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 08:57:41 (1786021061)
[ 5458.106960] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5460.080785] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 08:57:46 (1786021066)
[ 5461.681381] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5463.332659] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 08:57:49 (1786021069)
[ 5464.643360] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5467.020720] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 08:57:52 (1786021072)
[ 5468.664645] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5470.121543] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 08:57:56 (1786021076)
[ 5471.671200] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5473.960861] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 08:57:59 (1786021079)
[ 5476.780809] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5479.372685] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 08:58:04 (1786021084)
[ 5482.984389] Lustre: lustre-MDT0000-mdc-ffff90c8204dc800: Connection to lustre-MDT0000 (at 192.168.201.116@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5483.008422] Lustre: Skipped 13 previous similar messages
[ 5504.480352] Lustre: 2359:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786021095/real 1786021095]  req@ffff90c8048d4e00 x1872773216917248/t0(0) o400->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 224/224 e 0 to 1 dl 1786021111 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5504.509277] Lustre: 2359:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 14 previous similar messages
[ 5504.543836] Lustre: MGC192.168.201.116@tcp: Connection restored to 192.168.201.116@tcp (at 192.168.201.116@tcp)
[ 5504.561506] Lustre: Skipped 12 previous similar messages
[ 5523.239733] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 08:58:48 (1786021128)
[ 5554.406462] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 08:59:20 (1786021160)
[ 5555.323127] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5566.097880] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 08:59:31 (1786021171)
[ 5566.693449] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5577.015026] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 08:59:43 (1786021183)
[ 5577.520714] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5587.072320] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 08:59:53 (1786021193)
[ 5587.813449] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5598.777030] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 09:00:03 (1786021203)
[ 5600.430640] Lustre: *** cfs_fail_loc=510, val=3***
[ 5616.293124] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 09:00:20 (1786021220)
[ 5617.019387] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5631.054773] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 09:00:36 (1786021236)
[ 5631.517894] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5700.801650] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 09:01:46 (1786021306)
[ 5701.334670] LustreError: 83057:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5704.725098] LustreError: lustre-MDT0000-mdc-ffff90c8204dc800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5704.756337] LustreError: 83071:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff90c8204dc800: inode [0x20000a042:0x8:0x0] mdc close failed: rc = -108
[ 5704.804891] LustreError: 83071:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff90c8204dc800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff90c804a97a00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5705.408158] LustreError: 83057:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5707.531866] LustreError: 83087:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5710.610749] LustreError: lustre-MDT0000-mdc-ffff90c8204dc800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5710.632982] LustreError: 83101:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff90c8204dc800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff90c803b2be00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5710.655470] LustreError: 83101:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5711.632424] LustreError: 83087:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5718.849295] LustreError: lustre-MDT0000-mdc-ffff90c8204dc800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5719.334222] LustreError: 83143:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5722.405128] LustreError: lustre-MDT0000-mdc-ffff90c8204dc800: operation ldlm_enqueue to node 192.168.201.116@tcp failed: rc = -107
[ 5722.421945] LustreError: Skipped 3 previous similar messages
[ 5722.440679] LustreError: lustre-MDT0000-mdc-ffff90c8204dc800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5722.540714] LustreError: 83162:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5722.561964] LustreError: 83162:0:(file.c:6154:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5723.416218] LustreError: 83143:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5723.428764] LustreError: 83143:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff90c8204dc800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5723.701344] LustreError: 83169:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5723.712905] LustreError: 83169:0:(file.c:6154:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5726.060328] LustreError: 83191:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5726.074721] LustreError: 83191:0:(file.c:6154:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5729.744443] LustreError: 83218:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5732.883879] LustreError: lustre-MDT0000-mdc-ffff90c8204dc800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5732.915232] LustreError: 83232:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff90c8204dc800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff90c803b2bc00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5732.963543] LustreError: 83232:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5733.816250] LustreError: 83218:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5740.703852] LustreError: lustre-MDT0000-mdc-ffff90c8204dc800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5741.141403] LustreError: 83274:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5744.413527] LustreError: lustre-MDT0000-mdc-ffff90c8204dc800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5744.443283] LustreError: 83289:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff90c8204dc800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff90c805d27b00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5744.469215] LustreError: 83289:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5745.208519] LustreError: 83274:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5748.788390] LustreError: lustre-MDT0000-mdc-ffff90c8204dc800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5748.908666] LustreError: 83321:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5748.919074] LustreError: 83321:0:(file.c:6154:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5749.456879] LustreError: 83302:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff90c8204dc800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5758.774255] LustreError: 83374:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5758.799216] LustreError: 83374:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5762.792160] LustreError: 83374:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5762.805392] LustreError: 83374:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5764.254248] LustreError: lustre-MDT0000-mdc-ffff90c8204dc800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5773.851349] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 09:02:59 (1786021379)
[ 5793.943229] LustreError: 81089:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5798.032148] LustreError: 81089:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5808.069274] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 09:03:33 (1786021413)
[ 5861.685735] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 09:04:26 (1786021466)
[ 5930.418248] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 09:05:35 (1786021535)
[ 5968.669490] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 09:06:13 (1786021573)
[ 5969.324620] Lustre: Mounted lustre-client
[ 6093.073220] Lustre: Unmounted lustre-client
[ 6100.182592] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 09:08:26 (1786021706)
[ 6101.035193] LustreError: 87660:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 6103.912890] Lustre: lustre-OST0000-osc-ffff90c8204dc800: Connection to lustre-OST0000 (at 192.168.201.116@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6103.922943] Lustre: Skipped 13 previous similar messages
[ 6103.938483] LustreError: lustre-OST0000-osc-ffff90c8204dc800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6103.962336] LustreError: 87748:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff90c8204dc800: namespace resource [0x240000400:0x3f0d:0x0].0x0 (ffff90c805d27600) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6104.004385] LustreError: 87748:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 6104.224137] LustreError: 87660:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout interrupted
[ 6104.230988] Lustre: 2357:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.116@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 6110.314982] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 09:08:36 (1786021716)
[ 6167.520634] Lustre: 88327:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786021719/real 1786021719]  req@ffff90c80484bb80 x1872773217069568/t0(0) o101->lustre-MDT0000-mdc-ffff90c8204dc800@192.168.201.116@tcp:12/10 lens 328/344 e 0 to 1 dl 1786021774 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 6167.555345] Lustre: 88327:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
[ 6167.583373] Lustre: lustre-MDT0000-mdc-ffff90c8204dc800: Connection restored to 192.168.201.116@tcp (at 192.168.201.116@tcp)
[ 6167.590785] Lustre: Skipped 14 previous similar messages
[ 6178.770603] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 09:09:44 (1786021784)
[ 6180.550655] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 6182.831642] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 09:09:48 (1786021788)
[ 6250.036710] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6252.038851] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 09:10:57 (1786021857)
[ 6310.437223] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 09:11:56 (1786021916)
[ 6313.189944] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6315.078849] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 09:12:00 (1786021920)
[ 6316.712121] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6318.613044] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 09:12:04 (1786021924)
[ 6357.390468] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 09:12:43 (1786021963)
[ 6369.378923] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6395.872409] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 192.168.201.116@tcp) was lost; in progress operations using this service will fail
[ 6395.892231] LustreError: Skipped 2 previous similar messages
[ 6406.182387] Lustre: Evicted from MGS (at 192.168.201.116@tcp) after server handle changed from 0xfc433508c564bda6 to 0xfc433508c564d960
[ 6406.196464] Lustre: Skipped 2 previous similar messages
[ 6414.518891] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6416.687195] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6428.246034] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 09:13:53 (1786022033)
[ 6431.050393] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6433.096810] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 09:13:58 (1786022038)
[ 6447.085711] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 192.168.201.116@tcp) was lost; in progress operations using this service will fail
[ 6447.120857] Lustre: Evicted from MGS (at 192.168.201.116@tcp) after server handle changed from 0xfc433508c564d960 to 0xfc433508c564dafd
[ 6460.229734] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 09:14:26 (1786022066)
[ 6484.084706] Lustre: 68330:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.116@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6512.054244] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 09:15:18 (1786022118)
[ 6519.315425] LustreError: lustre-OST0000-osc-ffff90c8204dc800: operation ost_setattr to node 192.168.201.116@tcp failed: rc = -19
[ 6519.332213] LustreError: Skipped 6 previous similar messages
[ 6571.955391] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6575.491332] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6664.686038] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 192.168.201.116@tcp) was lost; in progress operations using this service will fail
[ 6664.701817] LustreError: Skipped 1 previous similar message
[ 6664.722593] Lustre: Evicted from MGS (at 192.168.201.116@tcp) after server handle changed from 0xfc433508c564dd88 to 0xfc433508c564ed6b
[ 6664.730624] Lustre: Skipped 1 previous similar message
[ 6675.238295] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6677.150334] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6715.625627] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6717.389168] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6874.352611] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 09:21:19 (1786022479)
[ 6886.437506] Lustre: lustre-OST0000-osc-ffff90c8204dc800: Connection to lustre-OST0000 (at 192.168.201.116@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6886.446850] Lustre: Skipped 9 previous similar messages
[ 6945.131650] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6948.425212] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 7211.238924] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 09:26:56 (1786022816)
[ 7417.328032] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 192.168.201.116@tcp) was lost; in progress operations using this service will fail
[ 7417.346979] LustreError: Skipped 1 previous similar message
[ 7417.371661] Lustre: Evicted from MGS (at 192.168.201.116@tcp) after server handle changed from 0xfc433508c564f004 to 0xfc433508c5744043
[ 7417.381710] Lustre: Skipped 1 previous similar message
[ 7417.396484] Lustre: MGC192.168.201.116@tcp: Connection restored to 192.168.201.116@tcp (at 192.168.201.116@tcp)
[ 7417.410377] Lustre: Skipped 12 previous similar messages
[ 7455.049143] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 09:31:01 (1786023061)
[ 7456.411864] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7458.224072] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 09:31:04 (1786023064)
[ 7460.334804] LustreError: lustre-MDT0000-mdc-ffff90c8204dc800: operation ldlm_enqueue to node 192.168.201.116@tcp failed: rc = -107
[ 7460.345792] LustreError: Skipped 197 previous similar messages
[ 7460.360139] LustreError: lustre-MDT0000-mdc-ffff90c8204dc800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7468.443485] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 09:31:14 (1786023074)
[ 7636.536376] Lustre: Evicted from lustre-OST0000_UUID (at 192.168.201.116@tcp) after server handle changed from 0xfc433508c5744cc9 to 0xfc433508c5744dd3
[ 7636.585199] LustreError: lustre-OST0000-osc-ffff90c8204dc800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7641.024379] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 09:34:05 (1786023245)
[ 7669.216199] Lustre: 2356:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786023256/real 1786023256]  req@ffff90c8040d7100 x1872773234411520/t0(0) o4->lustre-OST0000-osc-ffff90c8204dc800@192.168.201.116@tcp:6/4 lens 4584/448 e 0 to 1 dl 1786023276 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7669.242068] Lustre: 2356:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
[ 7669.252614] Lustre: lustre-OST0000-osc-ffff90c8204dc800: Connection to lustre-OST0000 (at 192.168.201.116@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7669.276856] Lustre: Skipped 3 previous similar messages
[ 7669.293267] Lustre: lustre-OST0000-osc-ffff90c8204dc800: Connection restored to 192.168.201.116@tcp (at 192.168.201.116@tcp)
[ 7669.302157] Lustre: Skipped 2 previous similar messages
[ 7688.324745] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 09:34:53 (1786023293)
[ 7689.579809] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7691.147178] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 09:34:57 (1786023297)
[ 7692.664264] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7694.414990] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 09:35:00 (1786023300)
[ 7731.578262] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 09:35:37 (1786023337)
[ 7776.740776] Lustre: 2357:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786023368/real 1786023368]  req@ffff90c818df4000 x1872773234566784/t0(0) o400->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 224/224 e 0 to 1 dl 1786023384 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7776.779702] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 192.168.201.116@tcp) was lost; in progress operations using this service will fail
[ 7776.851196] Lustre: Evicted from MGS (at 192.168.201.116@tcp) after server handle changed from 0xfc433508c5744043 to 0xfc433508c5747120
[ 7793.977457] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 09:36:39 (1786023399)
[ 7795.543585] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7797.354059] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 09:36:43 (1786023403)
[ 7799.699733] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7802.045897] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 09:36:47 (1786023407)
[ 7824.145581] Lustre: Unmounted lustre-client
[ 7827.347264] Lustre: Mounted lustre-client
[ 7831.543953] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7853.024373] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 192.168.201.116@tcp) was lost; in progress operations using this service will fail
[ 7863.283742] Lustre: Evicted from MGS (at 192.168.201.116@tcp) after server handle changed from 0xfc433508c5747a18 to 0xfc433508c5747b1b
[ 7863.303505] Lustre: MGC192.168.201.116@tcp: Connection restored to 192.168.201.116@tcp (at 192.168.201.116@tcp)
[ 7863.311701] Lustre: Skipped 2 previous similar messages
[ 7878.533424] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2