[ 1997.425063] LustreError: lustre-OST0000-osc-ffff962b484ed800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1997.460280] Lustre: 2357:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.119@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 2005.176350] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 15:32:07 (1787859127)
[ 2007.232584] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 2009.314857] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 15:32:11 (1787859131)
[ 2010.632811] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 2012.875290] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 15:32:15 (1787859135)
[ 2017.182372] LustreError: lustre-MDT0000-mdc-ffff962b484ed800: operation mds_close to node 192.168.202.119@tcp failed: rc = -19
[ 2017.201682] LustreError: Skipped 1 previous similar message
[ 2037.217994] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 192.168.202.119@tcp) was lost; in progress operations using this service will fail
[ 2037.242335] Lustre: Evicted from MGS (at 192.168.202.119@tcp) after server handle changed from 0xd3a759d7452c5c2c to 0xd3a759d7452c744a
[ 2139.886489] LustreError: lustre-MDT0000-mdc-ffff962b484ed800: operation mds_close to node 192.168.202.119@tcp failed: rc = -19
[ 2139.898597] LustreError: Skipped 1 previous similar message
[ 2158.623310] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 192.168.202.119@tcp) was lost; in progress operations using this service will fail
[ 2158.659946] Lustre: Evicted from MGS (at 192.168.202.119@tcp) after server handle changed from 0xd3a759d7452c744a to 0xd3a759d7452e0cb9
[ 2187.358648] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 15:35:09 (1787859309)
[ 2187.910331] Lustre: *** cfs_fail_loc=305, val=0***
[ 2187.912620] Lustre: Skipped 4 previous similar messages
[ 2228.195432] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 192.168.202.119@tcp) was lost; in progress operations using this service will fail
[ 2238.453957] Lustre: Evicted from MGS (at 192.168.202.119@tcp) after server handle changed from 0xd3a759d7452e0cb9 to 0xd3a759d7452e1396
[ 2255.890766] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2257.676227] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2267.041600] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 15:36:29 (1787859389)
[ 2284.519411] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 192.168.202.119@tcp) was lost; in progress operations using this service will fail
[ 2284.564804] Lustre: Evicted from MGS (at 192.168.202.119@tcp) after server handle changed from 0xd3a759d7452e1396 to 0xd3a759d7452e1819
[ 2284.610676] LustreError: lustre-MDT0000-mdc-ffff962b484ed800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2307.546294] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 15:37:10 (1787859430)
[ 2325.485385] LustreError: lustre-OST0000-osc-ffff962b484ed800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2342.634654] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 15:37:45 (1787859465)
[ 2358.531101] LustreError: lustre-MDT0000-mdc-ffff962b484ed800: operation mds_close to node 192.168.202.119@tcp failed: rc = -19
[ 2377.247287] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 192.168.202.119@tcp) was lost; in progress operations using this service will fail
[ 2387.523165] Lustre: Evicted from MGS (at 192.168.202.119@tcp) after server handle changed from 0xd3a759d7452e1819 to 0xd3a759d7452e5fe0
[ 2410.997539] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2413.648053] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2500.575544] Lustre: 2356:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787859608/real 1787859608]  req@ffff962b59ce3c00 x1874704179986176/t0(0) o400->MGC192.168.202.119@tcp@192.168.202.119@tcp:26/25 lens 224/224 e 0 to 1 dl 1787859624 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2500.632713] Lustre: 2356:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 16 previous similar messages
[ 2500.644186] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 192.168.202.119@tcp) was lost; in progress operations using this service will fail
[ 2510.831606] Lustre: Evicted from MGS (at 192.168.202.119@tcp) after server handle changed from 0xd3a759d7452e5fe0 to 0xd3a759d745300323
[ 2532.658931] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2535.575203] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2602.428677] Lustre: lustre-MDT0000-mdc-ffff962b484ed800: Connection to lustre-MDT0000 (at 192.168.202.119@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2602.450019] Lustre: Skipped 9 previous similar messages
[ 2622.435276] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 192.168.202.119@tcp) was lost; in progress operations using this service will fail
[ 2622.472948] Lustre: Evicted from MGS (at 192.168.202.119@tcp) after server handle changed from 0xd3a759d745300323 to 0xd3a759d74531a93e
[ 2622.493300] Lustre: MGC192.168.202.119@tcp: Connection restored to 192.168.202.119@tcp (at 192.168.202.119@tcp)
[ 2622.517562] Lustre: Skipped 15 previous similar messages
[ 2654.160863] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2657.370073] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2688.607694] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 15:43:30 (1787859810)
[ 2693.606697] LustreError: lustre-MDT0000-mdc-ffff962b484ed800: operation mds_reint to node 192.168.202.119@tcp failed: rc = -19
[ 2693.619972] LustreError: Skipped 3 previous similar messages
[ 2714.591382] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 192.168.202.119@tcp) was lost; in progress operations using this service will fail
[ 2724.777694] Lustre: Evicted from MGS (at 192.168.202.119@tcp) after server handle changed from 0xd3a759d74531a93e to 0xd3a759d745326ed4
[ 2737.898236] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2786.080829] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2826.235505] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2859.999393] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 192.168.202.119@tcp) was lost; in progress operations using this service will fail
[ 2860.003788] LustreError: Skipped 2 previous similar messages
[ 2870.192583] Lustre: Evicted from MGS (at 192.168.202.119@tcp) after server handle changed from 0xd3a759d74532cce4 to 0xd3a759d74532e819
[ 2870.217610] Lustre: Skipped 2 previous similar messages
[ 2885.891743] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2948.782852] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 3006.524797] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 3106.869258] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 15:50:29 (1787860229)
[ 3167.499460] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3170.195530] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3452.902994] LustreError: lustre-OST0000-osc-ffff962b484ed800: operation ldlm_cancel to node 192.168.202.119@tcp failed: rc = -19
[ 3452.926052] LustreError: Skipped 8 previous similar messages
[ 3452.938034] Lustre: lustre-OST0000-osc-ffff962b484ed800: Connection to lustre-OST0000 (at 192.168.202.119@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3452.969639] Lustre: Skipped 8 previous similar messages
[ 3481.368783] Lustre: lustre-OST0000-osc-ffff962b484ed800: Connection restored to 192.168.202.119@tcp (at 192.168.202.119@tcp)
[ 3481.381871] Lustre: Skipped 16 previous similar messages
[ 3501.894701] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3504.923518] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3831.922278] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3834.524893] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4079.711969] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 16:06:42 (1787861202)
[ 4096.994825] Lustre: 50612:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787861205/real 1787861205]  req@ffff962b5816b840 x1874704200849152/t0(0) o101->lustre-MDT0000-mdc-ffff962b484ed800@192.168.202.119@tcp:12/10 lens 576/1152 e 0 to 1 dl 1787861221 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 4097.028023] Lustre: 50612:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
[ 4097.036478] Lustre: lustre-MDT0000-mdc-ffff962b484ed800: Connection to lustre-MDT0000 (at 192.168.202.119@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4097.049893] Lustre: Skipped 1 previous similar message
[ 4097.093016] Lustre: lustre-MDT0000-mdc-ffff962b484ed800: Connection restored to 192.168.202.119@tcp (at 192.168.202.119@tcp)
[ 4097.111941] Lustre: Skipped 1 previous similar message
[ 4106.470899] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 16:07:08 (1787861228)
[ 4134.594342] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 16:07:36 (1787861256)
[ 4162.851443] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 16:08:05 (1787861285)
[ 4163.246930] Lustre: Mounted lustre-client - version 2.17.57_81_g80478ec
[ 4194.271769] Lustre: 2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787861302/real 1787861302]  req@ffff962b58204780 x1874704200866816/t0(0) o400->MGC192.168.202.119@tcp@192.168.202.119@tcp:26/25 lens 224/224 e 0 to 1 dl 1787861318 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4194.319752] Lustre: 2359:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 4194.339234] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 192.168.202.119@tcp) was lost; in progress operations using this service will fail
[ 4194.366910] LustreError: Skipped 3 previous similar messages
[ 4204.560906] Lustre: Evicted from MGS (at 192.168.202.119@tcp) after server handle changed from 0xd3a759d74534afc1 to 0xd3a759d745488e44
[ 4204.583569] Lustre: Skipped 3 previous similar messages
[ 4214.073398] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4216.015403] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4218.320470] Lustre: Unmounted lustre-client
[ 4224.981808] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 16:09:07 (1787861347)
[ 4349.923657] Lustre: 2358:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787861458/real 1787861458]  req@ffff962b43e90b40 x1874704200900736/t0(0) o4->lustre-OST0000-osc-ffff962b484ed800@192.168.202.119@tcp:6/4 lens 488/448 e 0 to 1 dl 1787861474 ref 2 fl Rpc:XQr/602/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4349.947030] Lustre: 2358:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 58 previous similar messages
[ 4525.502207] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 16:14:08 (1787861648)
[ 4576.642741] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 16:14:58 (1787861698)
[ 4579.591484] Lustre: Unmounted lustre-client
[ 4605.183444] Lustre: Mounted lustre-client - version 2.17.57_81_g80478ec
[ 4612.330756] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 16:15:34 (1787861734)
[ 4612.780556] LustreError: 56544:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4613.840716] LustreError: 56544:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4614.025467] Lustre: *** cfs_fail_loc=305, val=0***
[ 4638.688974] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 16:16:01 (1787861761)
[ 4639.280516] Lustre: Mounted lustre-client - version 2.17.57_81_g80478ec
[ 4641.463595] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4651.828482] Lustre: Unmounted lustre-client
[ 4659.323827] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 16:16:21 (1787861781)
[ 4747.933608] LustreError: lustre-MDT0000-mdc-ffff962b47008000: operation mds_reint to node 192.168.202.119@tcp failed: rc = -19
[ 4747.941241] LustreError: Skipped 3 previous similar messages
[ 4747.945822] Lustre: lustre-MDT0000-mdc-ffff962b47008000: Connection to lustre-MDT0000 (at 192.168.202.119@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4747.964794] Lustre: Skipped 23 previous similar messages
[ 4767.200154] Lustre: 2358:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787861875/real 1787861875]  req@ffff962b4699fc00 x1874704203565440/t0(0) o400->MGC192.168.202.119@tcp@192.168.202.119@tcp:26/25 lens 224/224 e 0 to 1 dl 1787861891 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4767.240857] Lustre: 2358:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 74 previous similar messages
[ 4767.258979] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 192.168.202.119@tcp) was lost; in progress operations using this service will fail
[ 4767.284450] Lustre: Evicted from MGS (at 192.168.202.119@tcp) after server handle changed from 0xd3a759d74548ab5c to 0xd3a759d7454b5f0c
[ 4767.301168] Lustre: MGC192.168.202.119@tcp: Connection restored to 192.168.202.119@tcp (at 192.168.202.119@tcp)
[ 4767.313036] Lustre: Skipped 24 previous similar messages
[ 4945.775941] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 16:21:08 (1787862068)
[ 4951.450252] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4965.951858] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 192.168.202.119@tcp) was lost; in progress operations using this service will fail
[ 4966.000070] Lustre: Evicted from MGS (at 192.168.202.119@tcp) after server handle changed from 0xd3a759d7454b5f0c to 0xd3a759d745502a89
[ 4966.080371] LustreError: lustre-MDT0000-mdc-ffff962b47008000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4989.194707] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 16:21:51 (1787862111)
[ 4989.773984] Lustre: Mounted lustre-client - version 2.17.57_81_g80478ec
[ 4997.753388] LustreError: lustre-OST0000-osc-ffff962b47008000: operation ldlm_enqueue to node 192.168.202.119@tcp failed: rc = -107
[ 4997.796319] LustreError: lustre-OST0000-osc-ffff962b47008000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5010.203873] Lustre: Unmounted lustre-client
[ 5018.583758] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 16:22:21 (1787862141)
[ 5025.931780] LustreError: lustre-MDT0000-mdc-ffff962b47008000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5025.948071] LustreError: 60940:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 5035.260290] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 16:22:37 (1787862157)
[ 5035.516780] LustreError: 61642:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 5040.607196] LustreError: 61642:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=0
[ 5040.622662] LustreError: 61642:0:(import.c:711:ptlrpc_connect_import_locked()) already connected
[ 5040.726290] LustreError: lustre-MDT0000-mdc-ffff962b47008000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5040.742136] LustreError: 61662:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 5040.827241] LustreError: 61667:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5041.954509] LustreError: 61673:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5041.975408] LustreError: 61673:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5044.368339] LustreError: 61695:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5044.395067] LustreError: 61695:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5049.229394] LustreError: 61740:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5049.246455] LustreError: 61740:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 5060.440272] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 16:23:02 (1787862182)
[ 5100.975170] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5102.819372] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5113.831917] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 16:23:56 (1787862236)
[ 5161.563079] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5163.674410] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5177.941652] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 16:24:59 (1787862299)
[ 5247.629854] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5250.322284] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5261.309591] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 16:26:23 (1787862383)
[ 5302.402198] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5303.919145] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5310.079448] Lustre: Unmounted lustre-client
[ 5371.563048] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5374.688071] Lustre: Mounted lustre-client - version 2.17.57_81_g80478ec
[ 5382.779333] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 16:28:25 (1787862505)
[ 5386.361251] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5388.381892] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 16:28:30 (1787862510)
[ 5424.011429] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 16:29:05 (1787862545)
[ 5425.591991] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5428.812559] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 16:29:10 (1787862550)
[ 5431.632284] Lustre: *** cfs_fail_loc=805, val=0***
[ 5431.732194] Lustre: Mounted lustre-client - version 2.17.57_81_g80478ec
[ 5438.706108] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5458.399186] Lustre: 2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787862566/real 1787862566]  req@ffff962b443830c0 x1874704205626496/t0(0) o400->lustre-MDT0000-mdc-ffff962b47062800@192.168.202.119@tcp:12/10 lens 224/224 e 0 to 1 dl 1787862582 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5458.432776] Lustre: 2359:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5458.444785] Lustre: lustre-MDT0000-mdc-ffff962b47062800: Connection to lustre-MDT0000 (at 192.168.202.119@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5458.456034] Lustre: Skipped 12 previous similar messages
[ 5458.459783] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 192.168.202.119@tcp) was lost; in progress operations using this service will fail
[ 5468.647942] Lustre: Evicted from MGS (at 192.168.202.119@tcp) after server handle changed from 0xd3a759d745503660 to 0xd3a759d74550387b
[ 5468.658937] Lustre: MGC192.168.202.119@tcp: Connection restored to 192.168.202.119@tcp (at 192.168.202.119@tcp)
[ 5468.667075] Lustre: Skipped 12 previous similar messages
[ 5469.815203] Lustre: 68313:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.119@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 5483.010361] LustreError: 2355:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff962b47062800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff962b45ba3f00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5485.752978] Lustre: Unmounted lustre-client
[ 5494.453938] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 16:30:17 (1787862617)
[ 5537.903364] LustreError: 2355:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff962b45aef840 x1874704205638656/t94489280516(94489280516) o101->lustre-MDT0000-mdc-ffff962b43f9d000@192.168.202.119@tcp:12/10 lens 664/608 e 0 to 0 dl 1787862677 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5544.420052] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5546.311453] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5555.733540] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 16:31:18 (1787862678)
[ 5557.680222] LustreError: lustre-OST0000-osc-ffff962b43f9d000: operation ost_write to node 192.168.202.119@tcp failed: rc = -107
[ 5557.689860] LustreError: Skipped 2 previous similar messages
[ 5557.712707] LustreError: lustre-OST0000-osc-ffff962b43f9d000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5557.726991] Lustre: 2356:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.119@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5557.774988] LustreError: 72856:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff962b43f9d000: namespace resource [0x240000400:0x3c82:0x0].0x0 (ffff962b468b1d00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5557.791094] LustreError: 72856:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5567.846519] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 16:31:30 (1787862690)
[ 5569.777796] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5571.509299] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 16:31:34 (1787862694)
[ 5573.503110] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5575.939769] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 16:31:38 (1787862698)
[ 5577.866984] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5580.058783] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 16:31:42 (1787862702)
[ 5581.981504] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5584.664775] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 16:31:46 (1787862706)
[ 5586.354813] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5588.163920] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 16:31:50 (1787862710)
[ 5589.887835] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5591.894778] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 16:31:54 (1787862714)
[ 5593.696352] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5595.815440] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 16:31:58 (1787862718)
[ 5597.412300] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5599.445809] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 16:32:02 (1787862722)
[ 5601.078958] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5602.927270] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 16:32:05 (1787862725)
[ 5605.098929] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5607.177412] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 16:32:09 (1787862729)
[ 5608.757149] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5611.081219] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 16:32:13 (1787862733)
[ 5615.804318] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5618.410085] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 16:32:20 (1787862740)
[ 5662.430571] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 16:33:04 (1787862784)
[ 5694.860612] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 16:33:37 (1787862817)
[ 5695.434488] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5707.348689] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 16:33:49 (1787862829)
[ 5707.846708] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5718.418187] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 16:34:00 (1787862840)
[ 5718.838826] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5728.184272] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 16:34:11 (1787862851)
[ 5728.819550] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5737.822867] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 16:34:20 (1787862860)
[ 5738.237861] Lustre: *** cfs_fail_loc=510, val=3***
[ 5746.321545] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 16:34:29 (1787862869)
[ 5746.664043] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5815.168727] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 16:35:37 (1787862937)
[ 5815.818655] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5883.617233] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 16:36:46 (1787863006)
[ 5884.342451] LustreError: 83079:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5888.013845] LustreError: lustre-MDT0000-mdc-ffff962b43f9d000: operation ldlm_enqueue to node 192.168.202.119@tcp failed: rc = -107
[ 5888.051942] LustreError: lustre-MDT0000-mdc-ffff962b43f9d000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5888.086021] LustreError: 83094:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff962b43f9d000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff962b46a66c00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5888.359141] LustreError: 83079:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5890.599440] LustreError: 83110:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5893.887811] LustreError: lustre-MDT0000-mdc-ffff962b43f9d000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5893.905671] LustreError: 83124:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff962b43f9d000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff962b45ba3400) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5893.946709] LustreError: 83124:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5894.672610] LustreError: 83110:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5902.075460] LustreError: lustre-MDT0000-mdc-ffff962b43f9d000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5902.616603] LustreError: 83166:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5905.591810] LustreError: lustre-MDT0000-mdc-ffff962b43f9d000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5905.640517] LustreError: 83185:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5905.646623] LustreError: 83185:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5906.679129] LustreError: 83166:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5906.695617] LustreError: 83166:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff962b43f9d000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5906.713464] LustreError: 83191:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5906.721095] LustreError: 83191:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5909.180634] LustreError: 83214:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5909.195795] LustreError: 83214:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5913.152908] LustreError: 83241:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5916.562730] LustreError: lustre-MDT0000-mdc-ffff962b43f9d000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5916.586728] LustreError: 83255:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff962b43f9d000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff962b46a66e00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5916.658247] LustreError: 83255:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5917.186801] LustreError: 83241:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5924.915420] LustreError: lustre-MDT0000-mdc-ffff962b43f9d000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5925.352501] LustreError: 83297:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5928.820518] LustreError: lustre-MDT0000-mdc-ffff962b43f9d000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5928.850825] LustreError: 83312:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff962b43f9d000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff962b44b44500) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5928.898497] LustreError: 83312:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5929.439177] LustreError: 83297:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5932.738129] LustreError: lustre-MDT0000-mdc-ffff962b43f9d000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5932.839509] LustreError: 83344:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5932.850885] LustreError: 83344:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5933.703770] LustreError: 83325:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff962b43f9d000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5942.050685] LustreError: 83396:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5942.069025] LustreError: 83396:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5945.941795] LustreError: lustre-MDT0000-mdc-ffff962b43f9d000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5946.036619] LustreError: 83419:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5946.049995] LustreError: 83419:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5946.104372] LustreError: 83396:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5946.111611] LustreError: 83396:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5946.120459] LustreError: 83396:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff962b43f9d000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5959.230831] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 16:38:01 (1787863081)
[ 5979.641536] LustreError: 68310:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5979.651563] LustreError: 68310:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5983.727119] LustreError: 68310:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5983.735964] LustreError: 68310:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5993.262301] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 16:38:35 (1787863115)
[ 6046.536990] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 16:39:29 (1787863169)
[ 6104.031187] Lustre: 85577:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787863173/real 1787863173]  req@ffff962b442d1e00 x1874704205770112/t0(0) o101->lustre-MDT0000-mdc-ffff962b43f9d000@192.168.202.119@tcp:12/10 lens 576/1152 e 0 to 1 dl 1787863228 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'stat.0' uid:0 gid:0 projid:0
[ 6104.070161] Lustre: 85577:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 21 previous similar messages
[ 6104.078493] Lustre: lustre-MDT0000-mdc-ffff962b43f9d000: Connection to lustre-MDT0000 (at 192.168.202.119@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6104.088041] Lustre: Skipped 17 previous similar messages
[ 6104.107051] Lustre: lustre-MDT0000-mdc-ffff962b43f9d000: Connection restored to 192.168.202.119@tcp (at 192.168.202.119@tcp)
[ 6104.118893] Lustre: Skipped 17 previous similar messages
[ 6112.673627] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 16:40:34 (1787863234)
[ 6147.863796] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 16:41:10 (1787863270)
[ 6148.933371] Lustre: Mounted lustre-client - version 2.17.57_81_g80478ec
[ 6271.716318] Lustre: Unmounted lustre-client
[ 6280.245436] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 16:43:22 (1787863402)
[ 6281.049626] LustreError: 87724:0:(osc_request.c:3177:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 6283.559608] LustreError: lustre-OST0000-osc-ffff962b43f9d000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6283.572434] LustreError: 87807:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff962b43f9d000: namespace resource [0x240000400:0x3cad:0x0].0x0 (ffff962b46a66900) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6283.604391] LustreError: 87807:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 6283.930113] LustreError: 87724:0:(osc_request.c:3177:osc_build_rpc()) cfs_fail_timeout interrupted
[ 6283.939403] Lustre: 2356:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.119@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 6289.815191] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 16:43:32 (1787863412)
[ 6354.316675] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 16:44:36 (1787863476)
[ 6355.710509] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 6357.352953] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 16:44:40 (1787863480)
[ 6420.544318] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6422.023060] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 16:45:44 (1787863544)
[ 6480.303963] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 16:46:43 (1787863603)
[ 6482.981428] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6484.721070] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 16:46:47 (1787863607)
[ 6486.416599] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6488.145462] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 16:46:50 (1787863610)
[ 6523.663196] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 16:47:26 (1787863646)
[ 6535.712273] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6561.759740] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 192.168.202.119@tcp) was lost; in progress operations using this service will fail
[ 6561.777412] LustreError: Skipped 2 previous similar messages
[ 6561.814522] Lustre: Evicted from MGS (at 192.168.202.119@tcp) after server handle changed from 0xd3a759d7455044d0 to 0xd3a759d745505cd9
[ 6561.826119] Lustre: Skipped 2 previous similar messages
[ 6577.159434] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6578.517696] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6588.943414] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 16:48:31 (1787863711)
[ 6591.556635] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6593.606697] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 16:48:36 (1787863716)
[ 6607.857179] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 192.168.202.119@tcp) was lost; in progress operations using this service will fail
[ 6607.899358] Lustre: Evicted from MGS (at 192.168.202.119@tcp) after server handle changed from 0xd3a759d745505cd9 to 0xd3a759d745506178
[ 6625.194163] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 16:49:07 (1787863747)
[ 6655.612656] Lustre: 68313:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.119@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6678.063453] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 16:50:00 (1787863800)
[ 6685.776893] LustreError: lustre-OST0000-osc-ffff962b43f9d000: operation ost_setattr to node 192.168.202.119@tcp failed: rc = -107
[ 6685.795319] LustreError: Skipped 13 previous similar messages
[ 6704.096682] Lustre: 2358:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787863773/real 1787863773]  req@ffff962b45aeda40 x1874704205869440/t0(0) o400->lustre-MDT0000-mdc-ffff962b43f9d000@192.168.202.119@tcp:12/10 lens 224/224 e 0 to 1 dl 1787863828 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6704.110867] Lustre: 2358:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 16 previous similar messages
[ 6742.551675] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6746.035745] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6831.071714] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 192.168.202.119@tcp) was lost; in progress operations using this service will fail
[ 6831.093211] LustreError: Skipped 1 previous similar message
[ 6841.324381] Lustre: Evicted from MGS (at 192.168.202.119@tcp) after server handle changed from 0xd3a759d745506385 to 0xd3a759d745507353
[ 6841.325779] Lustre: lustre-MDT0000-mdc-ffff962b43f9d000: Connection to lustre-MDT0000 (at 192.168.202.119@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6841.329280] Lustre: Skipped 1 previous similar message
[ 6841.335570] Lustre: Skipped 8 previous similar messages
[ 6841.339442] Lustre: MGC192.168.202.119@tcp: Connection restored to 192.168.202.119@tcp (at 192.168.202.119@tcp)
[ 6841.339451] Lustre: Skipped 9 previous similar messages
[ 6852.936194] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6854.124429] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6893.357400] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6895.120307] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 7035.683371] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 16:55:58 (1787864158)
[ 7116.243622] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 7120.016767] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 7394.738317] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 17:01:57 (1787864517)
[ 7527.404515] Lustre: lustre-MDT0000-mdc-ffff962b43f9d000: Connection to lustre-MDT0000 (at 192.168.202.119@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7527.420318] Lustre: Skipped 2 previous similar messages
[ 7542.777455] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 192.168.202.119@tcp) was lost; in progress operations using this service will fail
[ 7542.792718] LustreError: Skipped 1 previous similar message
[ 7542.836585] Lustre: Evicted from MGS (at 192.168.202.119@tcp) after server handle changed from 0xd3a759d745507601 to 0xd3a759d745601905
[ 7542.851565] Lustre: Skipped 1 previous similar message
[ 7542.862724] Lustre: MGC192.168.202.119@tcp: Connection restored to 192.168.202.119@tcp (at 192.168.202.119@tcp)
[ 7542.870969] Lustre: Skipped 3 previous similar messages
[ 7579.033416] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 17:05:01 (1787864701)
[ 7580.814398] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7582.651791] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 17:05:05 (1787864705)
[ 7584.888808] LustreError: lustre-MDT0000-mdc-ffff962b43f9d000: operation ldlm_enqueue to node 192.168.202.119@tcp failed: rc = -107
[ 7584.901217] LustreError: Skipped 222 previous similar messages
[ 7584.924165] LustreError: lustre-MDT0000-mdc-ffff962b43f9d000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7593.549876] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 17:05:16 (1787864716)
[ 7761.902082] LustreError: lustre-OST0000-osc-ffff962b43f9d000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7763.696864] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 17:08:06 (1787864886)
[ 7789.538826] Lustre: 2356:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787864893/real 1787864893]  req@ffff962b584d1e00 x1874704223552000/t0(0) o4->lustre-OST0000-osc-ffff962b43f9d000@192.168.202.119@tcp:6/4 lens 4584/448 e 0 to 1 dl 1787864913 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7789.573784] Lustre: 2356:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
[ 7807.696811] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 17:08:50 (1787864930)
[ 7809.468747] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7811.352889] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 17:08:53 (1787864933)
[ 7812.856877] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7814.687926] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 17:08:57 (1787864937)
[ 7853.048911] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 17:09:35 (1787864975)
[ 7891.955789] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 192.168.202.119@tcp) was lost; in progress operations using this service will fail
[ 7891.997412] Lustre: Evicted from MGS (at 192.168.202.119@tcp) after server handle changed from 0xd3a759d745601905 to 0xd3a759d745603fb6
[ 7892.021820] Lustre: Skipped 1 previous similar message
[ 7906.249942] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 17:10:29 (1787865029)
[ 7907.740699] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7909.411865] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 17:10:32 (1787865032)
[ 7910.682080] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7912.389774] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 17:10:35 (1787865035)
[ 7923.067859] Lustre: Unmounted lustre-client
[ 7924.691082] Lustre: Mounted lustre-client - version 2.17.57_81_g80478ec
[ 7927.581501] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7958.467872] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2