[ 1979.417577] LustreError: lustre-OST0000-osc-ffff91acc85d7000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1979.439496] Lustre: 2351:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.121@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1985.823536] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 07:24:03 (1787138643)
[ 1987.975160] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1989.685584] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 07:24:07 (1787138647)
[ 1991.418712] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1993.173458] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 07:24:11 (1787138651)
[ 2016.744856] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 192.168.201.121@tcp) was lost; in progress operations using this service will fail
[ 2026.986449] Lustre: Evicted from MGS (at 192.168.201.121@tcp) after server handle changed from 0x6be087856309707b to 0x6be0878563098bf6
[ 2130.460563] LustreError: lustre-MDT0000-mdc-ffff91acc85d7000: operation mds_reint to node 192.168.201.121@tcp failed: rc = -19
[ 2130.480383] LustreError: Skipped 2 previous similar messages
[ 2147.871655] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 192.168.201.121@tcp) was lost; in progress operations using this service will fail
[ 2158.105469] Lustre: Evicted from MGS (at 192.168.201.121@tcp) after server handle changed from 0x6be0878563098bf6 to 0x6be08785630b233f
[ 2173.775363] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 07:27:11 (1787138831)
[ 2174.409823] Lustre: *** cfs_fail_loc=305, val=0***
[ 2174.412667] Lustre: Skipped 2 previous similar messages
[ 2194.801751] LustreError: lustre-OST0001-osc-ffff91acc85d7000: operation ost_connect to node 192.168.201.121@tcp failed: rc = -75
[ 2221.536524] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 192.168.201.121@tcp) was lost; in progress operations using this service will fail
[ 2230.766824] Lustre: Evicted from MGS (at 192.168.201.121@tcp) after server handle changed from 0x6be08785630b233f to 0x6be08785630b2935
[ 2230.780724] Lustre: MGC192.168.201.121@tcp: Connection restored to 192.168.201.121@tcp (at 192.168.201.121@tcp)
[ 2230.797979] Lustre: Skipped 13 previous similar messages
[ 2232.772500] Lustre: lustre-MDT0000-mdc-ffff91acc85d7000: Connection to lustre-MDT0000 (at 192.168.201.121@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2232.786896] Lustre: Skipped 11 previous similar messages
[ 2247.799109] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2249.414590] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2258.558608] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 07:28:36 (1787138916)
[ 2271.738426] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 192.168.201.121@tcp) was lost; in progress operations using this service will fail
[ 2271.765315] Lustre: Evicted from MGS (at 192.168.201.121@tcp) after server handle changed from 0x6be08785630b2935 to 0x6be08785630b2cd8
[ 2273.441527] LustreError: lustre-MDT0000-mdc-ffff91acc85d7000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2300.259242] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 07:29:18 (1787138958)
[ 2322.419473] LustreError: lustre-OST0000-osc-ffff91acc85d7000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2341.898727] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 07:29:59 (1787138999)
[ 2356.761938] LustreError: lustre-MDT0000-mdc-ffff91acc85d7000: operation mds_reint to node 192.168.201.121@tcp failed: rc = -19
[ 2374.623220] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 192.168.201.121@tcp) was lost; in progress operations using this service will fail
[ 2383.798560] Lustre: Evicted from MGS (at 192.168.201.121@tcp) after server handle changed from 0x6be08785630b2cd8 to 0x6be08785630b7188
[ 2402.835539] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2405.767749] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2490.911726] Lustre: 2353:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787139134/real 1787139134]  req@ffff91acc7be0380 x1873948689098880/t0(0) o400->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 224/224 e 0 to 1 dl 1787139150 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2490.981961] Lustre: 2353:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 15 previous similar messages
[ 2491.009502] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 192.168.201.121@tcp) was lost; in progress operations using this service will fail
[ 2501.237664] Lustre: Evicted from MGS (at 192.168.201.121@tcp) after server handle changed from 0x6be08785630b7188 to 0x6be08785630cc2d8
[ 2522.339627] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2524.684578] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2609.123876] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 192.168.201.121@tcp) was lost; in progress operations using this service will fail
[ 2619.383836] Lustre: Evicted from MGS (at 192.168.201.121@tcp) after server handle changed from 0x6be08785630cc2d8 to 0x6be08785630e2109
[ 2639.814213] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2642.520942] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2672.079285] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 07:35:30 (1787139330)
[ 2677.512496] LustreError: lustre-MDT0000-mdc-ffff91acc85d7000: operation ldlm_enqueue to node 192.168.201.121@tcp failed: rc = -19
[ 2677.521057] LustreError: Skipped 3 previous similar messages
[ 2696.162209] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 192.168.201.121@tcp) was lost; in progress operations using this service will fail
[ 2706.420512] Lustre: Evicted from MGS (at 192.168.201.121@tcp) after server handle changed from 0x6be08785630e2109 to 0x6be08785630ede64
[ 2716.151351] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2760.549373] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2810.485440] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2844.640145] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 192.168.201.121@tcp) was lost; in progress operations using this service will fail
[ 2844.658868] LustreError: Skipped 2 previous similar messages
[ 2853.804944] Lustre: Evicted from MGS (at 192.168.201.121@tcp) after server handle changed from 0x6be08785630f2d08 to 0x6be08785630f7a39
[ 2853.813467] Lustre: Skipped 2 previous similar messages
[ 2853.840693] Lustre: MGC192.168.201.121@tcp: Connection restored to 192.168.201.121@tcp (at 192.168.201.121@tcp)
[ 2853.853336] Lustre: Skipped 16 previous similar messages
[ 2863.601298] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2887.989617] Lustre: lustre-MDT0000-mdc-ffff91acc85d7000: Connection to lustre-MDT0000 (at 192.168.201.121@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2888.007649] Lustre: Skipped 9 previous similar messages
[ 2916.021414] Lustre: 15959:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.121@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2927.360967] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2986.305533] Lustre: 15959:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.121@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 3001.087019] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 3062.960787] Lustre: 15959:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.121@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 3102.827553] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 07:42:40 (1787139760)
[ 3160.760916] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3163.350587] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3448.796238] LustreError: lustre-OST0000-osc-ffff91acc85d7000: operation ldlm_enqueue to node 192.168.201.121@tcp failed: rc = -19
[ 3448.815655] LustreError: Skipped 9 previous similar messages
[ 3492.207579] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3494.486594] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3780.482268] Lustre: lustre-OST0000-osc-ffff91acc85d7000: Connection to lustre-OST0000 (at 192.168.201.121@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3780.505521] Lustre: Skipped 4 previous similar messages
[ 3827.706590] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3830.261907] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4077.109336] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 07:58:54 (1787140734)
[ 4094.945124] Lustre: 50585:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787140738/real 1787140738]  req@ffff91acd06e6d80 x1873948708304128/t0(0) o101->lustre-MDT0000-mdc-ffff91acc85d7000@192.168.201.121@tcp:12/10 lens 576/1152 e 0 to 1 dl 1787140754 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 4094.981487] Lustre: 50585:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
[ 4095.034852] Lustre: lustre-MDT0000-mdc-ffff91acc85d7000: Connection restored to 192.168.201.121@tcp (at 192.168.201.121@tcp)
[ 4095.047468] Lustre: Skipped 8 previous similar messages
[ 4103.484650] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 07:59:21 (1787140761)
[ 4129.789697] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 07:59:47 (1787140787)
[ 4157.132968] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 08:00:14 (1787140814)
[ 4157.789428] Lustre: Mounted lustre-client - version 2.17.57_8_gdef02f8
[ 4189.665401] Lustre: 2352:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787140832/real 1787140832]  req@ffff91acd00ca680 x1873948708323328/t0(0) o400->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 224/224 e 0 to 1 dl 1787140848 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4189.697955] Lustre: 2352:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 4189.713924] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 192.168.201.121@tcp) was lost; in progress operations using this service will fail
[ 4189.729017] LustreError: Skipped 3 previous similar messages
[ 4199.940604] Lustre: Evicted from MGS (at 192.168.201.121@tcp) after server handle changed from 0x6be08785631159f8 to 0x6be0878563232975
[ 4199.949801] Lustre: Skipped 3 previous similar messages
[ 4199.963826] Lustre: MGC192.168.201.121@tcp: Connection restored to 192.168.201.121@tcp (at 192.168.201.121@tcp)
[ 4199.977649] Lustre: Skipped 2 previous similar messages
[ 4222.141688] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4224.571711] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4227.143287] Lustre: Unmounted lustre-client
[ 4235.753356] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 08:01:33 (1787140893)
[ 4347.873089] Lustre: 2353:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787140991/real 1787140991]  req@ffff91acd06e5180 x1873948708359936/t0(0) o4->lustre-OST0000-osc-ffff91acc85d7000@192.168.201.121@tcp:6/4 lens 488/448 e 0 to 1 dl 1787141007 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4347.899106] Lustre: 2353:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 45 previous similar messages
[ 4364.268857] Lustre: lustre-OST0000-osc-ffff91acc85d7000: Connection restored to 192.168.201.121@tcp (at 192.168.201.121@tcp)
[ 4364.281250] Lustre: Skipped 7 previous similar messages
[ 4380.639260] Lustre: lustre-OST0000-osc-ffff91acc85d7000: Connection to lustre-OST0000 (at 192.168.201.121@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4380.673793] Lustre: Skipped 12 previous similar messages
[ 4540.758057] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 08:06:38 (1787141198)
[ 4591.911731] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 08:07:29 (1787141249)
[ 4596.743192] Lustre: Unmounted lustre-client
[ 4624.982429] Lustre: Mounted lustre-client - version 2.17.57_8_gdef02f8
[ 4633.113830] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 08:08:10 (1787141290)
[ 4633.721171] LustreError: 56537:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4634.770746] LustreError: 56537:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4634.999680] Lustre: *** cfs_fail_loc=305, val=0***
[ 4659.272313] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 08:08:36 (1787141316)
[ 4659.758713] Lustre: Mounted lustre-client - version 2.17.57_8_gdef02f8
[ 4661.503754] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4671.838550] Lustre: Unmounted lustre-client
[ 4679.094125] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 08:08:57 (1787141337)
[ 4794.492228] LustreError: lustre-MDT0000-mdc-ffff91acc8039800: operation mds_reint to node 192.168.201.121@tcp failed: rc = -19
[ 4794.511744] LustreError: Skipped 1 previous similar message
[ 4815.327320] Lustre: 2352:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787141458/real 1787141458]  req@ffff91acd00a3800 x1873948711034240/t0(0) o400->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 224/224 e 0 to 1 dl 1787141474 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4815.365031] Lustre: 2352:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 91 previous similar messages
[ 4815.373763] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 192.168.201.121@tcp) was lost; in progress operations using this service will fail
[ 4825.574080] Lustre: Evicted from MGS (at 192.168.201.121@tcp) after server handle changed from 0x6be08785632346cc to 0x6be087856325fabb
[ 4825.604168] Lustre: MGC192.168.201.121@tcp: Connection restored to 192.168.201.121@tcp (at 192.168.201.121@tcp)
[ 4825.614788] Lustre: Skipped 12 previous similar messages
[ 5012.595711] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 08:14:30 (1787141670)
[ 5018.695754] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5033.455191] Lustre: lustre-MDT0000-mdc-ffff91acc8039800: Connection to lustre-MDT0000 (at 192.168.201.121@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5033.471929] Lustre: Skipped 12 previous similar messages
[ 5033.479857] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 192.168.201.121@tcp) was lost; in progress operations using this service will fail
[ 5033.504089] Lustre: Evicted from MGS (at 192.168.201.121@tcp) after server handle changed from 0x6be087856325fabb to 0x6be08785632ac76c
[ 5033.611389] LustreError: lustre-MDT0000-mdc-ffff91acc8039800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5054.213184] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 08:15:11 (1787141711)
[ 5054.633866] Lustre: Mounted lustre-client - version 2.17.57_8_gdef02f8
[ 5062.293647] LustreError: lustre-OST0000-osc-ffff91acc8039800: operation ldlm_enqueue to node 192.168.201.121@tcp failed: rc = -107
[ 5062.322446] LustreError: lustre-OST0000-osc-ffff91acc8039800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5075.557385] Lustre: Unmounted lustre-client
[ 5084.038865] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 08:15:41 (1787141741)
[ 5091.231557] LustreError: lustre-MDT0000-mdc-ffff91acc8039800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5091.250646] LustreError: 60934:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 5100.339803] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 08:15:57 (1787141757)
[ 5100.841124] LustreError: 61639:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 5105.106850] LustreError: lustre-MDT0000-mdc-ffff91acc8039800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5105.124116] LustreError: 61655:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 5105.135558] LustreError: 61639:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=713
[ 5105.153647] LustreError: 61639:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting
[ 5105.220280] LustreError: 61660:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5106.378079] LustreError: 61667:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5106.389273] LustreError: 61667:0:(file.c:6156:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5108.843851] LustreError: 61689:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5108.868094] LustreError: 61689:0:(file.c:6156:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5113.821140] LustreError: 61734:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5113.837643] LustreError: 61734:0:(file.c:6156:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 5125.343882] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 08:16:22 (1787141782)
[ 5174.096514] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5175.931225] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5187.562786] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 08:17:25 (1787141845)
[ 5233.962876] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5235.756695] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5250.465788] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 08:18:27 (1787141907)
[ 5323.465850] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5325.021275] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5334.946729] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 08:19:52 (1787141992)
[ 5380.115876] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5382.273175] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5390.042189] Lustre: Unmounted lustre-client
[ 5450.929648] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5453.859399] Lustre: Mounted lustre-client - version 2.17.57_8_gdef02f8
[ 5462.386220] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 08:21:59 (1787142119)
[ 5465.471240] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5467.611533] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 08:22:05 (1787142125)
[ 5504.982750] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 08:22:42 (1787142162)
[ 5506.941356] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5509.860889] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 08:22:47 (1787142167)
[ 5511.167212] Lustre: *** cfs_fail_loc=805, val=0***
[ 5511.275145] Lustre: Mounted lustre-client - version 2.17.57_8_gdef02f8
[ 5518.320319] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5538.271269] Lustre: 2353:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787142181/real 1787142181]  req@ffff91ace0027800 x1873948713100416/t0(0) o400->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 224/224 e 0 to 1 dl 1787142197 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5538.307643] Lustre: 2353:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5538.322557] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 192.168.201.121@tcp) was lost; in progress operations using this service will fail
[ 5548.520352] Lustre: Evicted from MGS (at 192.168.201.121@tcp) after server handle changed from 0x6be08785632ad1de to 0x6be08785632ad3f9
[ 5548.534124] Lustre: MGC192.168.201.121@tcp: Connection restored to 192.168.201.121@tcp (at 192.168.201.121@tcp)
[ 5548.541375] Lustre: Skipped 11 previous similar messages
[ 5562.872877] LustreError: 2349:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff91acc8fad800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff91acc8399c00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5567.199075] Lustre: Unmounted lustre-client
[ 5577.423530] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 08:23:54 (1787142234)
[ 5604.946700] LustreError: 2349:0:(client.c:3435:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff91ace0025180 x1873948713113984/t94489280516(94489280516) o101->lustre-MDT0000-mdc-ffff91acc8f94800@192.168.201.121@tcp:12/10 lens 648/608 e 0 to 0 dl 1787142280 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5621.302175] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5623.115931] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5632.287473] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 08:24:50 (1787142290)
[ 5634.261275] LustreError: lustre-OST0000-osc-ffff91acc8f94800: operation ost_write to node 192.168.201.121@tcp failed: rc = -107
[ 5634.270818] LustreError: Skipped 2 previous similar messages
[ 5634.275879] Lustre: lustre-OST0000-osc-ffff91acc8f94800: Connection to lustre-OST0000 (at 192.168.201.121@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5634.291744] Lustre: Skipped 13 previous similar messages
[ 5634.311780] LustreError: lustre-OST0000-osc-ffff91acc8f94800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5634.338867] Lustre: 2351:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.121@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5634.403280] LustreError: 72857:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff91acc8f94800: namespace resource [0x240000400:0x37e2:0x0].0x0 (ffff91acc8d7db00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5634.416874] LustreError: 72857:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5644.604782] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 08:25:01 (1787142301)
[ 5647.208946] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5649.664411] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 08:25:07 (1787142307)
[ 5651.913524] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5653.715085] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 08:25:11 (1787142311)
[ 5655.557047] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5657.878795] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 08:25:15 (1787142315)
[ 5659.628103] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5661.823314] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 08:25:19 (1787142319)
[ 5664.153397] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5666.519740] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 08:25:23 (1787142323)
[ 5668.556503] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5670.767913] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 08:25:28 (1787142328)
[ 5672.663301] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5675.016065] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 08:25:32 (1787142332)
[ 5676.696822] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5678.882596] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 08:25:36 (1787142336)
[ 5680.625394] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5682.753419] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 08:25:40 (1787142340)
[ 5684.396892] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5686.473065] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 08:25:44 (1787142344)
[ 5688.109959] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5689.983242] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 08:25:47 (1787142347)
[ 5692.359958] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5694.136879] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 08:25:52 (1787142352)
[ 5729.016544] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 08:26:27 (1787142387)
[ 5759.842251] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 08:26:57 (1787142417)
[ 5760.387439] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5772.201921] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 08:27:09 (1787142429)
[ 5772.467793] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5782.524159] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 08:27:20 (1787142440)
[ 5783.118069] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5792.153742] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 08:27:29 (1787142449)
[ 5792.547092] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5803.008370] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 08:27:40 (1787142460)
[ 5803.464841] Lustre: *** cfs_fail_loc=510, val=3***
[ 5814.005958] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 08:27:51 (1787142471)
[ 5814.840066] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5826.569636] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 08:28:03 (1787142483)
[ 5826.970952] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5895.744360] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 08:29:12 (1787142552)
[ 5896.212364] LustreError: 83067:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5899.133473] LustreError: lustre-MDT0000-mdc-ffff91acc8f94800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5899.180483] LustreError: 83081:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff91acc8f94800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff91acc8399d00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5900.303192] LustreError: 83067:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5902.662162] LustreError: 83097:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5905.885498] LustreError: lustre-MDT0000-mdc-ffff91acc8f94800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5905.902554] LustreError: 83111:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff91acc8f94800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff91acc6a69c00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5905.920898] LustreError: 83111:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5906.759671] LustreError: 83097:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5913.981724] LustreError: lustre-MDT0000-mdc-ffff91acc8f94800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5914.005370] LustreError: 83140:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff91acc8f94800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff91acc8399d00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5914.041597] LustreError: 83140:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5914.583154] LustreError: 83153:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5917.565373] LustreError: lustre-MDT0000-mdc-ffff91acc8f94800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5917.675952] LustreError: 83172:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5917.693111] LustreError: 83172:0:(file.c:6156:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5918.656091] LustreError: 83153:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5918.670870] LustreError: 83153:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff91acc8f94800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5918.818045] LustreError: 83179:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5918.824480] LustreError: 83179:0:(file.c:6156:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5921.169788] LustreError: 83201:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5921.188430] LustreError: 83201:0:(file.c:6156:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5924.985230] LustreError: 83228:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5927.877897] LustreError: lustre-MDT0000-mdc-ffff91acc8f94800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5927.889195] LustreError: 83242:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff91acc8f94800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff91acc6a69b00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5927.911857] LustreError: 83247:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5927.921935] LustreError: 83247:0:(file.c:6156:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5929.007349] LustreError: 83228:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5936.066758] LustreError: lustre-MDT0000-mdc-ffff91acc8f94800: operation ldlm_enqueue to node 192.168.201.121@tcp failed: rc = -107
[ 5936.073466] LustreError: Skipped 5 previous similar messages
[ 5936.095620] LustreError: lustre-MDT0000-mdc-ffff91acc8f94800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5936.518901] LustreError: 83284:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5939.621221] LustreError: lustre-MDT0000-mdc-ffff91acc8f94800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5940.607195] LustreError: 83284:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5943.790229] LustreError: lustre-MDT0000-mdc-ffff91acc8f94800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5943.906348] LustreError: 83331:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5944.855824] LustreError: 83312:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff91acc8f94800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5953.308367] LustreError: 83383:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5953.321027] LustreError: 83383:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5957.011433] LustreError: lustre-MDT0000-mdc-ffff91acc8f94800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5957.399196] LustreError: 83383:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5957.403698] LustreError: 83383:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5957.411045] LustreError: 83383:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff91acc8f94800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5960.533339] LustreError: 83435:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5960.547114] LustreError: 83435:0:(file.c:6156:ll_inode_revalidate_fini()) Skipped 46 previous similar messages
[ 5970.450758] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 08:30:27 (1787142627)
[ 5991.377053] LustreError: 68310:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5991.389226] LustreError: 68310:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5995.399407] LustreError: 68310:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5995.413567] LustreError: 68310:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 6003.046449] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 08:31:01 (1787142661)
[ 6056.259591] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 08:31:53 (1787142713)
[ 6122.266667] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 08:32:59 (1787142779)
[ 6156.817635] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 08:33:34 (1787142814)
[ 6157.412373] Lustre: Mounted lustre-client - version 2.17.57_8_gdef02f8
[ 6281.308262] Lustre: Unmounted lustre-client
[ 6289.888379] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 08:35:47 (1787142947)
[ 6290.995898] LustreError: 87712:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 6294.394694] Lustre: lustre-OST0000-osc-ffff91acc8f94800: Connection to lustre-OST0000 (at 192.168.201.121@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6294.419026] Lustre: Skipped 14 previous similar messages
[ 6294.445824] LustreError: lustre-OST0000-osc-ffff91acc8f94800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6294.461347] LustreError: 87794:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff91acc8f94800: namespace resource [0x240000400:0x380d:0x0].0x0 (ffff91acc6a69700) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6294.490403] LustreError: 87794:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 3 previous similar messages
[ 6294.935471] LustreError: 87712:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout interrupted
[ 6294.955716] Lustre: 2350:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.121@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 6304.173168] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 08:36:01 (1787142961)
[ 6362.079250] Lustre: 88380:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787142966/real 1787142966]  req@ffff91acd0791880 x1873948713289984/t0(0) o101->lustre-MDT0000-mdc-ffff91acc8f94800@192.168.201.121@tcp:12/10 lens 328/344 e 0 to 1 dl 1787143021 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 6362.103776] Lustre: 88380:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 21 previous similar messages
[ 6362.115069] Lustre: lustre-MDT0000-mdc-ffff91acc8f94800: Connection restored to 192.168.201.121@tcp (at 192.168.201.121@tcp)
[ 6362.131173] Lustre: Skipped 18 previous similar messages
[ 6371.005749] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 08:37:08 (1787143028)
[ 6373.208512] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 6375.716806] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 08:37:12 (1787143032)
[ 6440.127852] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6442.405969] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 08:38:19 (1787143099)
[ 6501.095925] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 08:39:18 (1787143158)
[ 6503.473741] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6505.989240] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 08:39:23 (1787143163)
[ 6507.687382] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6509.340402] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 08:39:27 (1787143167)
[ 6553.311726] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 08:40:10 (1787143210)
[ 6565.431361] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6592.480393] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 192.168.201.121@tcp) was lost; in progress operations using this service will fail
[ 6592.489363] LustreError: Skipped 2 previous similar messages
[ 6601.785373] Lustre: Evicted from MGS (at 192.168.201.121@tcp) after server handle changed from 0x6be08785632ae01d to 0x6be08785632afae2
[ 6601.801281] Lustre: Skipped 2 previous similar messages
[ 6611.364912] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6613.362369] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6625.646447] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 08:41:23 (1787143283)
[ 6629.309078] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6631.306312] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 08:41:28 (1787143288)
[ 6652.895989] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 192.168.201.121@tcp) was lost; in progress operations using this service will fail
[ 6652.933178] Lustre: Evicted from MGS (at 192.168.201.121@tcp) after server handle changed from 0x6be08785632afae2 to 0x6be08785632afcfd
[ 6667.707842] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 08:42:05 (1787143325)
[ 6723.880515] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 08:43:00 (1787143380)
[ 6733.001049] LustreError: lustre-OST0000-osc-ffff91acc8f94800: operation ost_setattr to node 192.168.201.121@tcp failed: rc = -19
[ 6733.017978] LustreError: Skipped 4 previous similar messages
[ 6792.928850] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6797.218343] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6883.297924] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 192.168.201.121@tcp) was lost; in progress operations using this service will fail
[ 6883.324539] LustreError: Skipped 1 previous similar message
[ 6893.556786] Lustre: Evicted from MGS (at 192.168.201.121@tcp) after server handle changed from 0x6be08785632b0006 to 0x6be08785632b0d6c
[ 6893.564783] Lustre: Skipped 1 previous similar message
[ 6910.022192] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6912.149723] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6946.425527] Lustre: 68313:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.121@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6948.721676] Lustre: lustre-MDT0000-mdc-ffff91acc8f94800: Connection to lustre-MDT0000 (at 192.168.201.121@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6948.728991] Lustre: Skipped 8 previous similar messages
[ 6965.611903] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6968.088738] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6979.615125] Lustre: 2352:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787143578/real 1787143578]  req@ffff91aced71aa00 x1873948719824768/t0(0) o400->lustre-MDT0000-mdc-ffff91acc8f94800@192.168.201.121@tcp:12/10 lens 224/224 e 0 to 1 dl 1787143638 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6979.652625] Lustre: 2352:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 25 previous similar messages
[ 7120.841568] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 08:49:38 (1787143778)
[ 7193.588433] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 7198.792492] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 7490.939473] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 08:55:48 (1787144148)
[ 7656.443458] Lustre: lustre-MDT0000-mdc-ffff91acc8f94800: Connection to lustre-MDT0000 (at 192.168.201.121@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7656.458662] Lustre: Skipped 1 previous similar message
[ 7671.785899] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 192.168.201.121@tcp) was lost; in progress operations using this service will fail
[ 7671.798312] LustreError: Skipped 1 previous similar message
[ 7671.813289] Lustre: Evicted from MGS (at 192.168.201.121@tcp) after server handle changed from 0x6be08785632b1163 to 0x6be08785633ab56a
[ 7671.835898] Lustre: Skipped 1 previous similar message
[ 7671.848155] Lustre: MGC192.168.201.121@tcp: Connection restored to 192.168.201.121@tcp (at 192.168.201.121@tcp)
[ 7671.858574] Lustre: Skipped 12 previous similar messages
[ 7706.055337] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 08:59:24 (1787144364)
[ 7708.217829] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7710.758320] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 08:59:27 (1787144367)
[ 7713.186169] LustreError: lustre-MDT0000-mdc-ffff91acc8f94800: operation ldlm_enqueue to node 192.168.201.121@tcp failed: rc = -107
[ 7713.197855] LustreError: Skipped 59 previous similar messages
[ 7713.224595] LustreError: lustre-MDT0000-mdc-ffff91acc8f94800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7713.245507] LustreError: 102080:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff91acc8f94800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff91acd8862b00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 7721.691765] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 08:59:39 (1787144379)
[ 7890.220918] Lustre: Evicted from lustre-OST0000_UUID (at 192.168.201.121@tcp) after server handle changed from 0x6be08785633aba4f to 0x6be08785633abb59
[ 7890.237514] LustreError: lustre-OST0000-osc-ffff91acc8f94800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7892.336364] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 09:02:29 (1787144549)
[ 7918.047509] Lustre: 2351:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787144557/real 1787144557]  req@ffff91acc7422680 x1873948731006976/t0(0) o4->lustre-OST0000-osc-ffff91acc8f94800@192.168.201.121@tcp:6/4 lens 4584/448 e 0 to 1 dl 1787144577 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7918.098859] Lustre: 2351:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
[ 7918.120182] Lustre: lustre-OST0000-osc-ffff91acc8f94800: Connection restored to 192.168.201.121@tcp (at 192.168.201.121@tcp)
[ 7918.131306] Lustre: Skipped 2 previous similar messages
[ 7936.705292] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 09:03:14 (1787144594)
[ 7938.592223] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7940.243950] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 09:03:18 (1787144598)
[ 7942.074846] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7943.886596] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 09:03:21 (1787144601)
[ 7978.811653] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 09:03:56 (1787144636)
[ 8023.526613] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 192.168.201.121@tcp) was lost; in progress operations using this service will fail
[ 8023.573856] Lustre: Evicted from MGS (at 192.168.201.121@tcp) after server handle changed from 0x6be08785633ab56a to 0x6be08785633adea6
[ 8037.477437] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 09:04:54 (1787144694)
[ 8039.444896] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 8041.506891] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 09:04:59 (1787144699)
[ 8043.127991] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 8044.414445] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 09:05:02 (1787144702)
[ 8061.498444] Lustre: Unmounted lustre-client
[ 8063.883334] Lustre: Mounted lustre-client - version 2.17.57_8_gdef02f8
[ 8067.288652] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 8090.591705] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 192.168.201.121@tcp) was lost; in progress operations using this service will fail
[ 8090.627797] Lustre: Evicted from MGS (at 192.168.201.121@tcp) after server handle changed from 0x6be08785633ae797 to 0x6be08785633ae7eb
[ 8090.663629] Lustre: MGC192.168.201.121@tcp: Connection restored to 192.168.201.121@tcp (at 192.168.201.121@tcp)
[ 8090.677978] Lustre: Skipped 2 previous similar messages
[ 8091.782573] Lustre: 106869:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.121@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 8108.796637] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2