[ 1895.541788] LustreError: lustre-OST0000-osc-ffff94cd8fdd3000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1895.557363] Lustre: 2355:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.206.106@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1902.516318] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 08:24:59 (1787660699)
[ 1905.174798] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1907.972458] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 08:25:03 (1787660703)
[ 1910.343213] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1912.646451] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 08:25:09 (1787660709)
[ 1934.815932] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 192.168.206.106@tcp) was lost; in progress operations using this service will fail
[ 1945.085093] Lustre: Evicted from MGS (at 192.168.206.106@tcp) after server handle changed from 0xd50253086c465be8 to 0xd50253086c46758e
[ 2040.164308] LustreError: lustre-MDT0000-mdc-ffff94cd8fdd3000: operation mds_reint to node 192.168.206.106@tcp failed: rc = -19
[ 2040.171775] LustreError: Skipped 2 previous similar messages
[ 2057.247315] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 192.168.206.106@tcp) was lost; in progress operations using this service will fail
[ 2067.434850] Lustre: Evicted from MGS (at 192.168.206.106@tcp) after server handle changed from 0xd50253086c46758e to 0xd50253086c4838c1
[ 2084.179377] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 08:28:00 (1787660880)
[ 2084.909272] Lustre: *** cfs_fail_loc=305, val=0***
[ 2084.914306] Lustre: Skipped 2 previous similar messages
[ 2104.645687] LustreError: lustre-OST0001-osc-ffff94cd8fdd3000: operation ost_connect to node 192.168.206.106@tcp failed: rc = -75
[ 2131.423459] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 192.168.206.106@tcp) was lost; in progress operations using this service will fail
[ 2141.163379] Lustre: Evicted from MGS (at 192.168.206.106@tcp) after server handle changed from 0xd50253086c4838c1 to 0xd50253086c483f43
[ 2146.246138] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2147.955791] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2156.033448] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 08:29:12 (1787660952)
[ 2171.883778] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 192.168.206.106@tcp) was lost; in progress operations using this service will fail
[ 2171.899276] Lustre: lustre-MDT0000-mdc-ffff94cd8fdd3000: Connection to lustre-MDT0000 (at 192.168.206.106@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2171.911518] Lustre: Skipped 12 previous similar messages
[ 2171.917179] Lustre: Evicted from MGS (at 192.168.206.106@tcp) after server handle changed from 0xd50253086c483f43 to 0xd50253086c4841dc
[ 2171.924891] Lustre: MGC192.168.206.106@tcp: Connection restored to 192.168.206.106@tcp (at 192.168.206.106@tcp)
[ 2171.929108] Lustre: Skipped 15 previous similar messages
[ 2171.977797] LustreError: lustre-MDT0000-mdc-ffff94cd8fdd3000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2189.483855] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 08:29:46 (1787660986)
[ 2204.934031] LustreError: lustre-OST0000-osc-ffff94cd8fdd3000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2223.733920] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 08:30:20 (1787661020)
[ 2237.380280] LustreError: lustre-MDT0000-mdc-ffff94cd8fdd3000: operation ldlm_enqueue to node 192.168.206.106@tcp failed: rc = -19
[ 2253.793313] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 192.168.206.106@tcp) was lost; in progress operations using this service will fail
[ 2263.979577] Lustre: Evicted from MGS (at 192.168.206.106@tcp) after server handle changed from 0xd50253086c4841dc to 0xd50253086c489167
[ 2281.475544] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2283.340597] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2366.431216] Lustre: 2355:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787661148/real 1787661148]  req@ffff94cd83fa1f80 x1874496183486208/t0(0) o400->MGC192.168.206.106@tcp@192.168.206.106@tcp:26/25 lens 224/224 e 0 to 1 dl 1787661164 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2366.466054] Lustre: 2355:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 13 previous similar messages
[ 2366.473541] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 192.168.206.106@tcp) was lost; in progress operations using this service will fail
[ 2376.689959] Lustre: Evicted from MGS (at 192.168.206.106@tcp) after server handle changed from 0xd50253086c489167 to 0xd50253086c4a1aa2
[ 2391.796956] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2393.749986] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2477.024297] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 192.168.206.106@tcp) was lost; in progress operations using this service will fail
[ 2487.270214] Lustre: Evicted from MGS (at 192.168.206.106@tcp) after server handle changed from 0xd50253086c4a1aa2 to 0xd50253086c4bba49
[ 2505.343702] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2507.347621] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2537.322976] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 08:35:33 (1787661333)
[ 2541.437923] LustreError: lustre-MDT0000-mdc-ffff94cd8fdd3000: operation ldlm_enqueue to node 192.168.206.106@tcp failed: rc = -19
[ 2541.447130] LustreError: Skipped 3 previous similar messages
[ 2557.919390] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 192.168.206.106@tcp) was lost; in progress operations using this service will fail
[ 2566.338106] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2568.108876] Lustre: Evicted from MGS (at 192.168.206.106@tcp) after server handle changed from 0xd50253086c4bba49 to 0xd50253086c4c7f45
[ 2590.764705] Lustre: 15933:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.206.106@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2604.758656] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2634.543509] Lustre: 15933:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.206.106@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2649.845480] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2691.781281] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2733.024163] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 192.168.206.106@tcp) was lost; in progress operations using this service will fail
[ 2733.038151] LustreError: Skipped 3 previous similar messages
[ 2743.272832] Lustre: Evicted from MGS (at 192.168.206.106@tcp) after server handle changed from 0xd50253086c4cfe48 to 0xd50253086c4dc0b9
[ 2743.297214] Lustre: Skipped 3 previous similar messages
[ 2754.068034] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2782.404534] Lustre: lustre-MDT0000-mdc-ffff94cd8fdd3000: Connection to lustre-MDT0000 (at 192.168.206.106@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2782.427758] Lustre: Skipped 8 previous similar messages
[ 2809.782928] Lustre: MGC192.168.206.106@tcp: Connection restored to 192.168.206.106@tcp (at 192.168.206.106@tcp)
[ 2809.795428] Lustre: Skipped 17 previous similar messages
[ 2811.122336] Lustre: 15933:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.206.106@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2821.141388] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2876.336406] Lustre: 15933:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.206.106@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2916.600347] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 08:41:53 (1787661713)
[ 2971.086598] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 2973.207199] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3261.411892] LustreError: lustre-OST0000-osc-ffff94cd8fdd3000: operation ldlm_enqueue to node 192.168.206.106@tcp failed: rc = -107
[ 3261.424801] LustreError: Skipped 10 previous similar messages
[ 3302.508750] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3304.551023] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3591.599117] Lustre: lustre-OST0000-osc-ffff94cd8fdd3000: Connection to lustre-OST0000 (at 192.168.206.106@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3591.619256] Lustre: Skipped 3 previous similar messages
[ 3611.799258] Lustre: lustre-OST0000-osc-ffff94cd8fdd3000: Connection restored to 192.168.206.106@tcp (at 192.168.206.106@tcp)
[ 3611.806031] Lustre: Skipped 5 previous similar messages
[ 3628.618204] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3630.708057] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3887.234982] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 08:58:04 (1787662684)
[ 3905.503153] Lustre: 50587:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787662687/real 1787662687]  req@ffff94cd8555a680 x1874496206197376/t0(0) o101->lustre-MDT0000-mdc-ffff94cd8fdd3000@192.168.206.106@tcp:12/10 lens 576/1152 e 0 to 1 dl 1787662703 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3905.539148] Lustre: 50587:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
[ 3912.802517] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 08:58:29 (1787662709)
[ 3939.398926] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 08:58:55 (1787662735)
[ 3964.350426] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 08:59:21 (1787662761)
[ 3964.946578] Lustre: Mounted lustre-client - version 2.17.57_1_g9ee77b6
[ 3996.639292] Lustre: 2357:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787662778/real 1787662778]  req@ffff94cd8567e680 x1874496206216704/t0(0) o400->lustre-MDT0000-mdc-ffff94cd8fdd3000@192.168.206.106@tcp:12/10 lens 224/224 e 0 to 1 dl 1787662794 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3996.712591] Lustre: 2357:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 3996.725969] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 192.168.206.106@tcp) was lost; in progress operations using this service will fail
[ 3996.744836] LustreError: Skipped 2 previous similar messages
[ 4006.381496] Lustre: Evicted from MGS (at 192.168.206.106@tcp) after server handle changed from 0xd50253086c4f3556 to 0xd50253086c6490d4
[ 4006.390945] Lustre: Skipped 2 previous similar messages
[ 4020.128969] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4021.703127] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4024.065581] Lustre: Unmounted lustre-client
[ 4030.369608] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 09:00:27 (1787662827)
[ 4151.776280] Lustre: 2357:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787662933/real 1787662933]  req@ffff94cd8567c380 x1874496206250624/t0(0) o4->lustre-OST0000-osc-ffff94cd8fdd3000@192.168.206.106@tcp:6/4 lens 488/448 e 0 to 1 dl 1787662949 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4151.815943] Lustre: 2357:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 58 previous similar messages
[ 4199.903289] Lustre: lustre-OST0000-osc-ffff94cd8fdd3000: Connection to lustre-OST0000 (at 192.168.206.106@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4199.927351] Lustre: Skipped 14 previous similar messages
[ 4216.328842] Lustre: lustre-OST0000-osc-ffff94cd8fdd3000: Connection restored to 192.168.206.106@tcp (at 192.168.206.106@tcp)
[ 4216.342267] Lustre: Skipped 16 previous similar messages
[ 4324.302780] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 09:05:21 (1787663121)
[ 4372.601330] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 09:06:09 (1787663169)
[ 4376.985557] Lustre: Unmounted lustre-client
[ 4401.574426] Lustre: Mounted lustre-client - version 2.17.57_1_g9ee77b6
[ 4408.943930] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 09:06:45 (1787663205)
[ 4409.356386] LustreError: 56529:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4410.413505] LustreError: 56529:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4410.627784] Lustre: *** cfs_fail_loc=305, val=0***
[ 4435.433749] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 09:07:11 (1787663231)
[ 4435.927095] Lustre: Mounted lustre-client - version 2.17.57_1_g9ee77b6
[ 4438.159686] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4448.483550] Lustre: Unmounted lustre-client
[ 4455.422735] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 09:07:32 (1787663252)
[ 4545.239956] LustreError: lustre-MDT0000-mdc-ffff94cd83d2a800: operation mds_reint to node 192.168.206.106@tcp failed: rc = -19
[ 4545.254070] LustreError: Skipped 1 previous similar message
[ 4563.935383] Lustre: 2356:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787663346/real 1787663346]  req@ffff94cd887c7800 x1874496208927616/t0(0) o400->MGC192.168.206.106@tcp@192.168.206.106@tcp:26/25 lens 224/224 e 0 to 1 dl 1787663362 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4563.970630] Lustre: 2356:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 77 previous similar messages
[ 4563.982607] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 192.168.206.106@tcp) was lost; in progress operations using this service will fail
[ 4574.199666] Lustre: Evicted from MGS (at 192.168.206.106@tcp) after server handle changed from 0xd50253086c64ad3d to 0xd50253086c676308
[ 4726.269492] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 09:12:03 (1787663523)
[ 4731.147576] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4747.255516] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 192.168.206.106@tcp) was lost; in progress operations using this service will fail
[ 4747.274866] Lustre: Evicted from MGS (at 192.168.206.106@tcp) after server handle changed from 0xd50253086c676308 to 0xd50253086c6c29ae
[ 4748.822216] LustreError: lustre-MDT0000-mdc-ffff94cd83d2a800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4764.170851] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 09:12:41 (1787663561)
[ 4764.673077] Lustre: Mounted lustre-client - version 2.17.57_1_g9ee77b6
[ 4771.890527] LustreError: lustre-OST0000-osc-ffff94cd83d2a800: operation ldlm_enqueue to node 192.168.206.106@tcp failed: rc = -107
[ 4771.909916] LustreError: lustre-OST0000-osc-ffff94cd83d2a800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4784.039883] Lustre: Unmounted lustre-client
[ 4790.621872] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 09:13:07 (1787663587)
[ 4797.448813] LustreError: lustre-MDT0000-mdc-ffff94cd83d2a800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4797.486312] LustreError: 60920:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4797.511747] LustreError: 60920:0:(file.c:6156:ll_inode_revalidate_fini()) Skipped 1 previous similar message
[ 4805.756601] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 09:13:22 (1787663602)
[ 4805.981042] Lustre: lustre-MDT0000-mdc-ffff94cd83d2a800: Connection to lustre-MDT0000 (at 192.168.206.106@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4806.009050] Lustre: Skipped 15 previous similar messages
[ 4806.018055] LustreError: 61620:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4810.109641] LustreError: lustre-MDT0000-mdc-ffff94cd83d2a800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4810.121734] LustreError: 61642:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4810.139532] LustreError: 61620:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=904
[ 4810.161702] LustreError: 61620:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting
[ 4810.206203] LustreError: 61647:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4811.280577] LustreError: 61654:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4811.299741] LustreError: 61654:0:(file.c:6156:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4813.556765] LustreError: 61676:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4813.578757] LustreError: 61676:0:(file.c:6156:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4818.279676] LustreError: 61720:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4818.307461] LustreError: 61720:0:(file.c:6156:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4820.454565] Lustre: lustre-MDT0000-mdc-ffff94cd83d2a800: Connection restored to 192.168.206.106@tcp (at 192.168.206.106@tcp)
[ 4820.464643] Lustre: Skipped 16 previous similar messages
[ 4827.327733] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 09:13:44 (1787663624)
[ 4873.933111] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4875.604195] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4886.334445] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 09:14:43 (1787663683)
[ 4924.441056] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4925.961407] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4937.341564] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 09:15:33 (1787663733)
[ 5003.189344] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5004.481332] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5012.281295] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 09:16:49 (1787663809)
[ 5049.491049] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5051.415919] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5057.553532] Lustre: Unmounted lustre-client
[ 5111.983060] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5114.586831] Lustre: Mounted lustre-client - version 2.17.57_1_g9ee77b6
[ 5121.649908] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 09:18:38 (1787663918)
[ 5124.410933] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5126.635940] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 09:18:43 (1787663923)
[ 5156.324728] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 09:19:13 (1787663953)
[ 5157.550425] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5159.075903] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 09:19:16 (1787663956)
[ 5160.324523] Lustre: *** cfs_fail_loc=805, val=0***
[ 5160.368810] Lustre: Mounted lustre-client - version 2.17.57_1_g9ee77b6
[ 5164.753691] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5186.015497] Lustre: 2356:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787663968/real 1787663968]  req@ffff94cd8759f480 x1874496210978048/t0(0) o400->MGC192.168.206.106@tcp@192.168.206.106@tcp:26/25 lens 224/224 e 0 to 1 dl 1787663984 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5186.029883] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 192.168.206.106@tcp) was lost; in progress operations using this service will fail
[ 5186.052575] Lustre: 2356:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5186.085841] Lustre: Evicted from MGS (at 192.168.206.106@tcp) after server handle changed from 0xd50253086c6c35fc to 0xd50253086c6c391a
[ 5194.654232] Lustre: Unmounted lustre-client
[ 5202.765324] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 09:19:59 (1787663999)
[ 5246.458416] LustreError: 2353:0:(client.c:3435:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff94cd8759ed80 x1874496210989184/t90194313220(90194313220) o101->lustre-MDT0000-mdc-ffff94cd8fdd4000@192.168.206.106@tcp:12/10 lens 664/608 e 0 to 0 dl 1787664060 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5252.373576] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5253.777489] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5261.802867] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 09:20:58 (1787664058)
[ 5262.962992] LustreError: lustre-OST0000-osc-ffff94cd8fdd4000: operation ost_write to node 192.168.206.106@tcp failed: rc = -107
[ 5262.969569] LustreError: Skipped 1 previous similar message
[ 5262.985500] LustreError: lustre-OST0000-osc-ffff94cd8fdd4000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5263.004307] Lustre: 2356:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.206.106@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5263.043499] LustreError: 72825:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff94cd8fdd4000: namespace resource [0x240000400:0x3fa2:0x0].0x0 (ffff94cd886e8000) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5270.849249] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 09:21:07 (1787664067)
[ 5272.266231] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5274.021939] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 09:21:10 (1787664070)
[ 5275.241961] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5276.823892] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 09:21:13 (1787664073)
[ 5278.230613] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5279.895203] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 09:21:16 (1787664076)
[ 5281.438373] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5283.641434] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 09:21:20 (1787664080)
[ 5285.244653] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5286.567793] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 09:21:23 (1787664083)
[ 5288.221903] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5289.716962] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 09:21:26 (1787664086)
[ 5291.189881] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5292.909313] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 09:21:29 (1787664089)
[ 5294.407486] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5296.163374] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 09:21:32 (1787664092)
[ 5297.417598] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5298.770704] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 09:21:35 (1787664095)
[ 5299.891771] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5301.745845] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 09:21:38 (1787664098)
[ 5303.082714] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5304.599388] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 09:21:41 (1787664101)
[ 5307.252715] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5308.762826] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 09:21:45 (1787664105)
[ 5338.913482] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 09:22:15 (1787664135)
[ 5367.850727] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 09:22:44 (1787664164)
[ 5368.378222] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5378.026261] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 09:22:54 (1787664174)
[ 5378.431828] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5388.364462] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 09:23:05 (1787664185)
[ 5388.607787] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5396.262119] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 09:23:13 (1787664193)
[ 5396.601642] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5406.772266] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 09:23:23 (1787664203)
[ 5407.221753] Lustre: *** cfs_fail_loc=510, val=3***
[ 5416.938763] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 09:23:33 (1787664213)
[ 5417.473125] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5427.601924] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 09:23:44 (1787664224)
[ 5427.980238] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5488.098511] Lustre: lustre-OST0000-osc-ffff94cd8fdd4000: Connection to lustre-OST0000 (at 192.168.206.106@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5488.110947] Lustre: Skipped 10 previous similar messages
[ 5494.371920] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 09:24:50 (1787664290)
[ 5494.769833] LustreError: 83032:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5497.526923] LustreError: lustre-MDT0000-mdc-ffff94cd8fdd4000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5497.542985] LustreError: 83046:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff94cd8fdd4000: inode [0x20000a042:0x8:0x0] mdc close failed: rc = -108
[ 5497.584943] LustreError: 83046:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff94cd8fdd4000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff94cd83ffaa00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5497.617869] Lustre: lustre-MDT0000-mdc-ffff94cd8fdd4000: Connection restored to 192.168.206.106@tcp (at 192.168.206.106@tcp)
[ 5497.625487] Lustre: Skipped 10 previous similar messages
[ 5498.848051] LustreError: 83032:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5500.951857] LustreError: 83062:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5503.786116] LustreError: lustre-MDT0000-mdc-ffff94cd8fdd4000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5503.804932] LustreError: 83078:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff94cd8fdd4000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff94cd886e8e00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5503.834967] LustreError: 83078:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5505.023119] LustreError: 83062:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5511.982625] LustreError: lustre-MDT0000-mdc-ffff94cd8fdd4000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5512.000565] LustreError: 83107:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff94cd8fdd4000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff94cd886e8c00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5512.029373] LustreError: 83107:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5512.520858] LustreError: 83120:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5515.351725] LustreError: lustre-MDT0000-mdc-ffff94cd8fdd4000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5515.416592] LustreError: 83139:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5515.425749] LustreError: 83139:0:(file.c:6156:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5516.508064] LustreError: 83146:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5516.519149] LustreError: 83146:0:(file.c:6156:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5516.593914] LustreError: 83120:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5516.606644] LustreError: 83120:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff94cd8fdd4000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5518.756963] LustreError: 83168:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5518.769263] LustreError: 83168:0:(file.c:6156:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5522.306557] LustreError: 83195:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5525.122118] LustreError: lustre-MDT0000-mdc-ffff94cd8fdd4000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5525.136376] LustreError: 83209:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff94cd8fdd4000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff94cd887ec700) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5526.375584] LustreError: 83195:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5533.146618] LustreError: lustre-MDT0000-mdc-ffff94cd8fdd4000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5533.493732] LustreError: 83251:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5536.296443] LustreError: lustre-MDT0000-mdc-ffff94cd8fdd4000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5536.324835] LustreError: 83266:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff94cd8fdd4000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff94cd886e8400) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5536.359797] LustreError: 83266:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5537.567160] LustreError: 83251:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5540.380310] LustreError: lustre-MDT0000-mdc-ffff94cd8fdd4000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5540.472042] LustreError: 83298:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5540.484547] LustreError: 83298:0:(file.c:6156:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5541.720199] LustreError: 83279:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff94cd8fdd4000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5549.325457] LustreError: 83350:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5549.333824] LustreError: 83350:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5552.704333] LustreError: lustre-MDT0000-mdc-ffff94cd8fdd4000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5552.752523] LustreError: 83373:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5552.759969] LustreError: 83373:0:(file.c:6156:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5553.407261] LustreError: 83350:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5553.410930] LustreError: 83350:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5553.414912] LustreError: 83350:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff94cd8fdd4000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5563.237374] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 09:26:00 (1787664360)
[ 5581.351808] LustreError: 79784:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5581.365203] LustreError: 79784:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5585.431288] LustreError: 79784:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5585.441796] LustreError: 79784:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5590.383597] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 09:26:27 (1787664387)
[ 5639.341205] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 09:27:16 (1787664436)
[ 5702.863717] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 09:28:19 (1787664499)
[ 5735.488761] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 09:28:52 (1787664532)
[ 5735.917588] Lustre: Mounted lustre-client - version 2.17.57_1_g9ee77b6
[ 5859.459122] Lustre: Unmounted lustre-client
[ 5866.230809] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 09:31:03 (1787664663)
[ 5866.791187] LustreError: 87677:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5869.407591] LustreError: lustre-OST0000-osc-ffff94cd8fdd4000: operation ost_statfs to node 192.168.206.106@tcp failed: rc = -107
[ 5869.413226] LustreError: Skipped 9 previous similar messages
[ 5869.422347] LustreError: lustre-OST0000-osc-ffff94cd8fdd4000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5869.436871] LustreError: 87765:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff94cd8fdd4000: namespace resource [0x240000400:0x3fcd:0x0].0x0 (ffff94cd887af000) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5869.454338] LustreError: 87765:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5869.623690] LustreError: 87677:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5869.631928] Lustre: 2356:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.206.106@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5875.846773] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 09:31:12 (1787664672)
[ 5933.027608] Lustre: 88344:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787664676/real 1787664676]  req@ffff94cd84f15880 x1874496211156608/t0(0) o101->lustre-MDT0000-mdc-ffff94cd8fdd4000@192.168.206.106@tcp:12/10 lens 328/344 e 0 to 1 dl 1787664731 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 5933.067033] Lustre: 88344:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 16 previous similar messages
[ 5940.451604] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 09:32:17 (1787664737)
[ 5942.363901] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 5943.949349] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 09:32:20 (1787664740)
[ 6007.305609] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6008.660201] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 09:33:25 (1787664805)
[ 6066.794584] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 09:34:23 (1787664863)
[ 6068.840391] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6070.238448] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 09:34:27 (1787664867)
[ 6071.607080] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6073.344765] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 09:34:30 (1787664870)
[ 6107.889933] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 09:35:04 (1787664904)
[ 6120.378224] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6147.040256] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 192.168.206.106@tcp) was lost; in progress operations using this service will fail
[ 6147.051842] LustreError: Skipped 2 previous similar messages
[ 6147.062777] Lustre: lustre-MDT0000-mdc-ffff94cd8fdd4000: Connection to lustre-MDT0000 (at 192.168.206.106@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6147.070893] Lustre: Evicted from MGS (at 192.168.206.106@tcp) after server handle changed from 0xd50253086c6c445e to 0xd50253086c6c5d86
[ 6147.085801] Lustre: Skipped 16 previous similar messages
[ 6147.098956] Lustre: Skipped 2 previous similar messages
[ 6147.103746] Lustre: MGC192.168.206.106@tcp: Connection restored to 192.168.206.106@tcp (at 192.168.206.106@tcp)
[ 6147.115632] Lustre: Skipped 15 previous similar messages
[ 6159.377301] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6160.674346] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6169.953355] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 09:36:07 (1787664967)
[ 6172.304418] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6173.741462] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 09:36:10 (1787664970)
[ 6188.031454] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 192.168.206.106@tcp) was lost; in progress operations using this service will fail
[ 6188.069610] Lustre: Evicted from MGS (at 192.168.206.106@tcp) after server handle changed from 0xd50253086c6c5d86 to 0xd50253086c6c613e
[ 6197.701567] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 09:36:34 (1787664994)
[ 6220.677893] Lustre: 68287:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.206.106@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6243.229189] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 09:37:20 (1787665040)
[ 6300.392303] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6304.850058] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6393.567498] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 192.168.206.106@tcp) was lost; in progress operations using this service will fail
[ 6393.576978] LustreError: Skipped 1 previous similar message
[ 6393.589549] Lustre: Evicted from MGS (at 192.168.206.106@tcp) after server handle changed from 0xd50253086c6c634b to 0xd50253086c6c719f
[ 6393.598752] Lustre: Skipped 1 previous similar message
[ 6394.741090] Lustre: 68287:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.206.106@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6418.524690] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6420.212390] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6456.941627] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6458.654797] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6594.794132] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 09:43:11 (1787665391)
[ 6652.547405] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6655.965980] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6904.994775] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 09:48:22 (1787665702)
[ 7036.397226] Lustre: lustre-MDT0000-mdc-ffff94cd8fdd4000: Connection to lustre-MDT0000 (at 192.168.206.106@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7036.414281] Lustre: Skipped 6 previous similar messages
[ 7036.435305] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 192.168.206.106@tcp) was lost; in progress operations using this service will fail
[ 7036.459964] LustreError: Skipped 1 previous similar message
[ 7036.526699] Lustre: Evicted from MGS (at 192.168.206.106@tcp) after server handle changed from 0xd50253086c6c76ae to 0xd50253086c7c19a4
[ 7036.543060] Lustre: Skipped 1 previous similar message
[ 7036.554164] Lustre: MGC192.168.206.106@tcp: Connection restored to 192.168.206.106@tcp (at 192.168.206.106@tcp)
[ 7036.569673] Lustre: Skipped 9 previous similar messages
[ 7067.487263] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 09:51:04 (1787665864)
[ 7069.066593] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7071.259882] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 09:51:07 (1787665867)
[ 7073.468833] LustreError: lustre-MDT0000-mdc-ffff94cd8fdd4000: operation ldlm_enqueue to node 192.168.206.106@tcp failed: rc = -107
[ 7073.492992] LustreError: Skipped 6 previous similar messages
[ 7073.516198] LustreError: lustre-MDT0000-mdc-ffff94cd8fdd4000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7081.310332] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 09:51:18 (1787665878)
[ 7248.441335] LustreError: lustre-OST0000-osc-ffff94cd8fdd4000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7249.979309] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 09:54:07 (1787666047)
[ 7274.976498] Lustre: 2356:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787666052/real 1787666052]  req@ffff94cd96831500 x1874496228851712/t0(0) o4->lustre-OST0000-osc-ffff94cd8fdd4000@192.168.206.106@tcp:6/4 lens 4584/448 e 0 to 1 dl 1787666072 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7275.004744] Lustre: 2356:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 17 previous similar messages
[ 7292.351839] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 09:54:49 (1787666089)
[ 7293.910452] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7295.803883] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 09:54:52 (1787666092)
[ 7297.375918] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7299.088750] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 09:54:55 (1787666095)
[ 7332.305579] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 09:55:29 (1787666129)
[ 7370.216039] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 192.168.206.106@tcp) was lost; in progress operations using this service will fail
[ 7370.241957] Lustre: Evicted from MGS (at 192.168.206.106@tcp) after server handle changed from 0xd50253086c7c19a4 to 0xd50253086c7c409b
[ 7370.260487] Lustre: Skipped 1 previous similar message
[ 7384.088684] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 09:56:21 (1787666181)
[ 7385.407818] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7386.906331] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 09:56:24 (1787666184)
[ 7388.205121] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7389.695199] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 09:56:26 (1787666186)
[ 7406.914482] Lustre: Unmounted lustre-client
[ 7408.988587] Lustre: Mounted lustre-client - version 2.17.57_1_g9ee77b6
[ 7411.879612] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7430.624186] Lustre: 2357:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787666212/real 1787666212]  req@ffff94cd82f0e680 x1874496229038848/t0(0) o400->MGC192.168.206.106@tcp@192.168.206.106@tcp:26/25 lens 224/224 e 0 to 1 dl 1787666228 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7441.277357] Lustre: 106792:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.206.106@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 7453.974247] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2