[ 1966.021362] LustreError: lustre-OST0000-osc-ffff95dbd8906000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1966.042173] Lustre: 2358:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.103@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1973.736941] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 06:45:47 (1786963547)
[ 1975.664654] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1977.304496] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 06:45:51 (1786963551)
[ 1978.892023] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1980.554417] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 06:45:54 (1786963554)
[ 2005.408319] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 192.168.204.103@tcp) was lost; in progress operations using this service will fail
[ 2005.442157] Lustre: Evicted from MGS (at 192.168.204.103@tcp) after server handle changed from 0xeb1468f73de52f to 0xeb1468f73dfeab
[ 2107.308204] LustreError: lustre-MDT0000-mdc-ffff95dbd8906000: operation mds_reint to node 192.168.204.103@tcp failed: rc = -19
[ 2107.332717] LustreError: Skipped 4 previous similar messages
[ 2126.305102] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 192.168.204.103@tcp) was lost; in progress operations using this service will fail
[ 2136.493728] Lustre: Evicted from MGS (at 192.168.204.103@tcp) after server handle changed from 0xeb1468f73dfeab to 0xeb1468f73fcff3
[ 2153.867932] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 06:48:47 (1786963727)
[ 2154.182695] Lustre: *** cfs_fail_loc=305, val=0***
[ 2154.184789] Lustre: Skipped 2 previous similar messages
[ 2192.864475] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 192.168.204.103@tcp) was lost; in progress operations using this service will fail
[ 2202.095683] Lustre: Evicted from MGS (at 192.168.204.103@tcp) after server handle changed from 0xeb1468f73fcff3 to 0xeb1468f73fd350
[ 2203.329416] Lustre: 15929:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.204.103@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2205.893891] Lustre: lustre-MDT0000-mdc-ffff95dbd8906000: Connection to lustre-MDT0000 (at 192.168.204.103@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2205.912587] Lustre: Skipped 11 previous similar messages
[ 2206.092918] Lustre: lustre-MDT0000-mdc-ffff95dbd8906000: Connection restored to 192.168.204.103@tcp (at 192.168.204.103@tcp)
[ 2206.111665] Lustre: Skipped 14 previous similar messages
[ 2220.349465] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2221.990825] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2233.128026] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 06:50:06 (1786963806)
[ 2253.281204] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 192.168.204.103@tcp) was lost; in progress operations using this service will fail
[ 2253.298880] Lustre: Evicted from MGS (at 192.168.204.103@tcp) after server handle changed from 0xeb1468f73fd350 to 0xeb1468f73fda3b
[ 2253.410826] LustreError: lustre-MDT0000-mdc-ffff95dbd8906000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2271.569405] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 06:50:44 (1786963844)
[ 2285.133338] LustreError: lustre-OST0000-osc-ffff95dbd8906000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2307.423421] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 06:51:20 (1786963880)
[ 2321.032151] LustreError: lustre-MDT0000-mdc-ffff95dbd8906000: operation ldlm_enqueue to node 192.168.204.103@tcp failed: rc = -19
[ 2340.321423] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 192.168.204.103@tcp) was lost; in progress operations using this service will fail
[ 2350.515613] Lustre: Evicted from MGS (at 192.168.204.103@tcp) after server handle changed from 0xeb1468f73fda3b to 0xeb1468f7402543
[ 2369.238176] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2371.489085] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2456.047768] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 192.168.204.103@tcp) was lost; in progress operations using this service will fail
[ 2456.126871] Lustre: Evicted from MGS (at 192.168.204.103@tcp) after server handle changed from 0xeb1468f7402543 to 0xeb1468f741ebfd
[ 2474.986900] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2477.353672] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2543.357400] LustreError: lustre-MDT0000-mdc-ffff95dbd8906000: operation ldlm_enqueue to node 192.168.204.103@tcp failed: rc = -19
[ 2543.370260] LustreError: Skipped 1 previous similar message
[ 2563.562152] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 192.168.204.103@tcp) was lost; in progress operations using this service will fail
[ 2563.597616] Lustre: Evicted from MGS (at 192.168.204.103@tcp) after server handle changed from 0xeb1468f741ebfd to 0xeb1468f743ce0f
[ 2579.044566] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2581.337487] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2611.775859] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 06:56:25 (1786964185)
[ 2635.233876] Lustre: 2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786964194/real 1786964194]  req@ffff95dbc814c000 x1873765130496000/t0(0) o400->MGC192.168.204.103@tcp@192.168.204.103@tcp:26/25 lens 224/224 e 0 to 1 dl 1786964210 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2635.272601] Lustre: 2358:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 20 previous similar messages
[ 2635.282413] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 192.168.204.103@tcp) was lost; in progress operations using this service will fail
[ 2645.498351] Lustre: Evicted from MGS (at 192.168.204.103@tcp) after server handle changed from 0xeb1468f743ce0f to 0xeb1468f744b811
[ 2646.584992] Lustre: 15929:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.204.103@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2658.402817] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2691.195670] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2733.892123] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2766.304354] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 192.168.204.103@tcp) was lost; in progress operations using this service will fail
[ 2766.322432] LustreError: Skipped 2 previous similar messages
[ 2776.553488] Lustre: Evicted from MGS (at 192.168.204.103@tcp) after server handle changed from 0xeb1468f744e250 to 0xeb1468f7454eec
[ 2776.558887] Lustre: Skipped 2 previous similar messages
[ 2784.822391] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2807.980779] LustreError: lustre-MDT0000-mdc-ffff95dbd8906000: operation ldlm_enqueue to node 192.168.204.103@tcp failed: rc = -19
[ 2807.989301] Lustre: lustre-MDT0000-mdc-ffff95dbd8906000: Connection to lustre-MDT0000 (at 192.168.204.103@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2807.992845] LustreError: Skipped 4 previous similar messages
[ 2808.026720] Lustre: Skipped 8 previous similar messages
[ 2827.761592] Lustre: MGC192.168.204.103@tcp: Connection restored to 192.168.204.103@tcp (at 192.168.204.103@tcp)
[ 2827.770929] Lustre: Skipped 16 previous similar messages
[ 2836.469993] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2893.747533] Lustre: 15929:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.204.103@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2905.605756] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2959.994113] Lustre: 15929:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.204.103@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 3000.630421] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 07:02:54 (1786964574)
[ 3058.761155] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3061.020580] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3346.269926] LustreError: lustre-OST0000-osc-ffff95dbd8906000: operation ost_write to node 192.168.204.103@tcp failed: rc = -19
[ 3346.282263] LustreError: Skipped 5 previous similar messages
[ 3384.980573] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3387.511812] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3677.831311] Lustre: lustre-OST0000-osc-ffff95dbd8906000: Connection to lustre-OST0000 (at 192.168.204.103@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3677.854866] Lustre: Skipped 4 previous similar messages
[ 3701.496872] Lustre: lustre-OST0000-osc-ffff95dbd8906000: Connection restored to 192.168.204.103@tcp (at 192.168.204.103@tcp)
[ 3701.508871] Lustre: Skipped 7 previous similar messages
[ 3722.704055] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3725.007976] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3973.434558] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 07:19:06 (1786965546)
[ 3991.009408] Lustre: 50580:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786965549/real 1786965549]  req@ffff95dbc8b6c380 x1873765149866368/t0(0) o101->lustre-MDT0000-mdc-ffff95dbd8906000@192.168.204.103@tcp:12/10 lens 576/1152 e 0 to 1 dl 1786965565 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3991.054174] Lustre: 50580:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
[ 4000.336062] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 07:19:33 (1786965573)
[ 4026.327612] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 07:19:59 (1786965599)
[ 4053.864930] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 07:20:27 (1786965627)
[ 4054.477654] Lustre: Mounted lustre-client - version 2.17.56_50_g3eb67fe
[ 4085.218131] Lustre: 2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786965644/real 1786965644]  req@ffff95dbc8b6df80 x1873765149886208/t0(0) o400->lustre-MDT0000-mdc-ffff95dbd8906000@192.168.204.103@tcp:12/10 lens 224/224 e 0 to 1 dl 1786965660 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4085.218675] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 192.168.204.103@tcp) was lost; in progress operations using this service will fail
[ 4085.286048] Lustre: 2358:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 4085.342965] LustreError: Skipped 3 previous similar messages
[ 4096.003102] Lustre: Evicted from MGS (at 192.168.204.103@tcp) after server handle changed from 0xeb1468f74713c3 to 0xeb1468f75b5e56
[ 4096.019473] Lustre: Skipped 3 previous similar messages
[ 4108.972703] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4111.166194] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4113.272895] Lustre: Unmounted lustre-client
[ 4121.319779] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 07:21:34 (1786965694)
[ 4244.448755] Lustre: 2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786965803/real 1786965803]  req@ffff95dbc629d180 x1873765149921024/t0(0) o4->lustre-OST0000-osc-ffff95dbd8906000@192.168.204.103@tcp:6/4 lens 488/448 e 0 to 1 dl 1786965819 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4244.499881] Lustre: 2358:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 53 previous similar messages
[ 4292.576791] Lustre: lustre-OST0000-osc-ffff95dbd8906000: Connection to lustre-OST0000 (at 192.168.204.103@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4292.591545] Lustre: Skipped 14 previous similar messages
[ 4309.018870] Lustre: lustre-OST0000-osc-ffff95dbd8906000: Connection restored to 192.168.204.103@tcp (at 192.168.204.103@tcp)
[ 4309.026117] Lustre: Skipped 16 previous similar messages
[ 4419.768895] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 07:26:32 (1786965992)
[ 4470.381612] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 07:27:23 (1786966043)
[ 4474.958695] Lustre: Unmounted lustre-client
[ 4497.513665] Lustre: Mounted lustre-client - version 2.17.56_50_g3eb67fe
[ 4503.687389] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 07:27:57 (1786966077)
[ 4504.075258] LustreError: 56511:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4505.124491] LustreError: 56511:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4505.378776] Lustre: *** cfs_fail_loc=305, val=0***
[ 4529.631122] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 07:28:23 (1786966103)
[ 4530.369952] Lustre: Mounted lustre-client - version 2.17.56_50_g3eb67fe
[ 4532.616883] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4542.936706] Lustre: Unmounted lustre-client
[ 4550.851476] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 07:28:44 (1786966124)
[ 4645.098077] LustreError: lustre-MDT0000-mdc-ffff95dbe00f2000: operation mds_reint to node 192.168.204.103@tcp failed: rc = -19
[ 4645.114334] LustreError: Skipped 1 previous similar message
[ 4665.824253] Lustre: 2359:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786966224/real 1786966224]  req@ffff95dbc4329180 x1873765152594432/t0(0) o400->MGC192.168.204.103@tcp@192.168.204.103@tcp:26/25 lens 224/224 e 0 to 1 dl 1786966240 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4665.873765] Lustre: 2359:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 82 previous similar messages
[ 4665.880281] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 192.168.204.103@tcp) was lost; in progress operations using this service will fail
[ 4665.896838] Lustre: Evicted from MGS (at 192.168.204.103@tcp) after server handle changed from 0xeb1468f75b772a to 0xeb1468f75e2879
[ 4667.321544] Lustre: 55906:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.204.103@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 4833.061891] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 07:33:26 (1786966406)
[ 4837.518138] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4853.229272] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 192.168.204.103@tcp) was lost; in progress operations using this service will fail
[ 4853.263607] Lustre: Evicted from MGS (at 192.168.204.103@tcp) after server handle changed from 0xeb1468f75e2879 to 0xeb1468f762f887
[ 4853.365769] LustreError: lustre-MDT0000-mdc-ffff95dbe00f2000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4867.267333] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 07:34:00 (1786966440)
[ 4867.758851] Lustre: Mounted lustre-client - version 2.17.56_50_g3eb67fe
[ 4874.987857] LustreError: lustre-OST0000-osc-ffff95dbe00f2000: operation ldlm_enqueue to node 192.168.204.103@tcp failed: rc = -107
[ 4874.998992] LustreError: lustre-OST0000-osc-ffff95dbe00f2000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4875.008861] LustreError: 60239:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff95dbe00f2000: namespace resource [0x240000400:0x3dc3:0x0].0x0 (ffff95dbc49b0100) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4886.619801] Lustre: Unmounted lustre-client
[ 4893.067326] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 07:34:26 (1786966466)
[ 4896.785674] Lustre: lustre-MDT0000-mdc-ffff95dbe00f2000: Connection to lustre-MDT0000 (at 192.168.204.103@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4896.798161] Lustre: Skipped 13 previous similar messages
[ 4899.966835] LustreError: lustre-MDT0000-mdc-ffff95dbe00f2000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4899.983299] LustreError: 60910:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4907.488419] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 07:34:41 (1786966481)
[ 4907.653768] LustreError: 61610:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4909.057825] Lustre: lustre-MDT0000-mdc-ffff95dbe00f2000: Connection restored to 192.168.204.103@tcp (at 192.168.204.103@tcp)
[ 4909.067877] Lustre: Skipped 15 previous similar messages
[ 4911.552993] LustreError: lustre-MDT0000-mdc-ffff95dbe00f2000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4911.569060] LustreError: 61631:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4911.580970] LustreError: 61610:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=1077
[ 4911.587828] LustreError: 61610:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting
[ 4911.676477] LustreError: 61637:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4912.767818] LustreError: 61643:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4912.779637] LustreError: 61643:0:(file.c:6154:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4915.060093] LustreError: 61665:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4915.078884] LustreError: 61665:0:(file.c:6154:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4919.659648] LustreError: 61710:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4919.668431] LustreError: 61710:0:(file.c:6154:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4928.323625] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 07:35:02 (1786966502)
[ 4966.782522] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4968.558585] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4977.957761] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 07:35:51 (1786966551)
[ 5015.129891] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5016.736747] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5026.661955] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 07:36:40 (1786966600)
[ 5093.156979] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5094.584563] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5102.629001] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 07:37:56 (1786966676)
[ 5141.711988] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5143.797971] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5150.381858] Lustre: Unmounted lustre-client
[ 5208.332511] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5211.726092] Lustre: Mounted lustre-client - version 2.17.56_50_g3eb67fe
[ 5219.618406] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 07:39:52 (1786966792)
[ 5222.658971] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5224.514951] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 07:39:57 (1786966797)
[ 5255.912582] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 07:40:29 (1786966829)
[ 5258.032655] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5260.476502] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 07:40:33 (1786966833)
[ 5262.217891] Lustre: *** cfs_fail_loc=805, val=0***
[ 5262.264206] Lustre: Mounted lustre-client - version 2.17.56_50_g3eb67fe
[ 5267.546299] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5288.737856] Lustre: 2359:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786966847/real 1786966847]  req@ffff95dbc816fb80 x1873765154651520/t0(0) o400->lustre-MDT0000-mdc-ffff95dbc2f6e800@192.168.204.103@tcp:12/10 lens 224/224 e 0 to 1 dl 1786966863 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5288.773948] Lustre: 2359:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5288.788088] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 192.168.204.103@tcp) was lost; in progress operations using this service will fail
[ 5298.161868] Lustre: Evicted from MGS (at 192.168.204.103@tcp) after server handle changed from 0xeb1468f76302ba to 0xeb1468f76304d5
[ 5313.535843] LustreError: 2357:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff95dbc6230000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff95dbc805cb00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5316.899236] Lustre: Unmounted lustre-client
[ 5325.906831] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 07:41:39 (1786966899)
[ 5353.975086] LustreError: 2357:0:(client.c:3435:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff95dbc816c000 x1873765154663424/t90194313220(90194313220) o101->lustre-MDT0000-mdc-ffff95dbc2f6e800@192.168.204.103@tcp:12/10 lens 664/608 e 0 to 0 dl 1786966944 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5364.046627] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5365.612652] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5374.392789] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 07:42:28 (1786966948)
[ 5376.159687] LustreError: lustre-OST0000-osc-ffff95dbc2f6e800: operation ost_write to node 192.168.204.103@tcp failed: rc = -107
[ 5376.177557] LustreError: Skipped 2 previous similar messages
[ 5376.203864] LustreError: lustre-OST0000-osc-ffff95dbc2f6e800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5376.219378] Lustre: 2360:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.103@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5376.255769] LustreError: 72816:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff95dbc2f6e800: namespace resource [0x240000400:0x3e22:0x0].0x0 (ffff95dbc6fbdb00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5376.273065] LustreError: 72816:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5384.273574] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 07:42:37 (1786966957)
[ 5386.011560] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5387.822424] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 07:42:41 (1786966961)
[ 5389.233208] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5391.352799] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 07:42:44 (1786966964)
[ 5393.281478] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5395.385044] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 07:42:48 (1786966968)
[ 5397.339199] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5399.173742] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 07:42:52 (1786966972)
[ 5400.547795] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5402.197128] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 07:42:55 (1786966975)
[ 5403.934890] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5405.957123] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 07:42:59 (1786966979)
[ 5407.716784] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5409.901870] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 07:43:03 (1786966983)
[ 5411.576673] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5413.799831] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 07:43:06 (1786966986)
[ 5415.573881] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5417.874631] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 07:43:11 (1786966991)
[ 5419.742384] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5421.599691] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 07:43:15 (1786966995)
[ 5423.469778] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5425.312864] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 07:43:18 (1786966998)
[ 5428.780469] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5430.952629] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 07:43:24 (1786967004)
[ 5464.384531] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 07:43:57 (1786967037)
[ 5494.613682] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 07:44:28 (1786967068)
[ 5495.426255] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5505.619220] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 07:44:38 (1786967078)
[ 5506.098850] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5516.872729] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 07:44:50 (1786967090)
[ 5517.237126] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5526.720918] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 07:45:00 (1786967100)
[ 5526.981707] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5538.398920] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 07:45:11 (1786967111)
[ 5538.902893] Lustre: *** cfs_fail_loc=510, val=3***
[ 5548.840920] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 07:45:22 (1786967122)
[ 5549.270929] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5560.405248] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 07:45:33 (1786967133)
[ 5560.792586] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5619.680298] Lustre: lustre-OST0000-osc-ffff95dbc2f6e800: Connection to lustre-OST0000 (at 192.168.204.103@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5619.702039] Lustre: Skipped 12 previous similar messages
[ 5626.336353] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 07:46:40 (1786967200)
[ 5626.672671] LustreError: 83025:0:(ldlm_flock.c:820:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5629.766042] LustreError: lustre-MDT0000-mdc-ffff95dbc2f6e800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5629.813422] LustreError: 83040:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff95dbc2f6e800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff95dbc805c500) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5629.878457] Lustre: lustre-MDT0000-mdc-ffff95dbc2f6e800: Connection restored to 192.168.204.103@tcp (at 192.168.204.103@tcp)
[ 5629.895688] Lustre: Skipped 11 previous similar messages
[ 5630.745262] LustreError: 83025:0:(ldlm_flock.c:820:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5632.891319] LustreError: 83055:0:(ldlm_flock.c:820:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5635.917549] LustreError: lustre-MDT0000-mdc-ffff95dbc2f6e800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5635.933287] LustreError: 83070:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff95dbc2f6e800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff95dbc49b0c00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5635.967538] LustreError: 83070:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5635.981384] LustreError: 83075:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5635.994882] LustreError: 83075:0:(file.c:6154:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5636.993527] LustreError: 83055:0:(ldlm_flock.c:820:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5644.035422] LustreError: lustre-MDT0000-mdc-ffff95dbc2f6e800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5644.721440] LustreError: 83111:0:(ldlm_flock.c:825:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5647.691768] LustreError: lustre-MDT0000-mdc-ffff95dbc2f6e800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5647.776490] LustreError: 83130:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5648.816195] LustreError: 83111:0:(ldlm_flock.c:825:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5648.833775] LustreError: 83111:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff95dbc2f6e800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5650.099453] LustreError: 83148:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5650.120152] LustreError: 83148:0:(file.c:6154:ll_inode_revalidate_fini()) Skipped 12 previous similar messages
[ 5655.014472] LustreError: 83185:0:(ldlm_flock.c:825:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5657.825547] LustreError: lustre-MDT0000-mdc-ffff95dbc2f6e800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5657.840201] LustreError: 83200:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff95dbc2f6e800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff95dbc8280500) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5657.864370] LustreError: 83200:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5657.892400] LustreError: 83205:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5657.914523] LustreError: 83205:0:(file.c:6154:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5659.112157] LustreError: 83185:0:(ldlm_flock.c:825:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5666.021582] LustreError: lustre-MDT0000-mdc-ffff95dbc2f6e800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5666.416298] LustreError: 83242:0:(ldlm_flock.c:874:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5669.201806] LustreError: lustre-MDT0000-mdc-ffff95dbc2f6e800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5670.488143] LustreError: 83242:0:(ldlm_flock.c:874:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5673.399292] LustreError: lustre-MDT0000-mdc-ffff95dbc2f6e800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5673.469765] LustreError: 83289:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5674.657250] LustreError: 83270:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff95dbc2f6e800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5682.711628] LustreError: 83341:0:(ldlm_flock.c:820:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5682.717946] LustreError: 83341:0:(ldlm_flock.c:820:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5686.142717] LustreError: lustre-MDT0000-mdc-ffff95dbc2f6e800: operation ldlm_enqueue to node 192.168.204.103@tcp failed: rc = -107
[ 5686.156234] LustreError: Skipped 8 previous similar messages
[ 5686.180616] LustreError: lustre-MDT0000-mdc-ffff95dbc2f6e800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5686.784112] LustreError: 83341:0:(ldlm_flock.c:820:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5686.794226] LustreError: 83341:0:(ldlm_flock.c:820:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5686.812246] LustreError: 83341:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff95dbc2f6e800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5689.742788] LustreError: 83392:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5689.756502] LustreError: 83392:0:(file.c:6154:ll_inode_revalidate_fini()) Skipped 46 previous similar messages
[ 5698.563181] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 07:47:51 (1786967271)
[ 5718.830969] LustreError: 68274:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5718.837964] LustreError: 68274:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5722.904124] LustreError: 68274:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5722.921372] LustreError: 68274:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5731.026931] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 07:48:24 (1786967304)
[ 5781.386971] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 07:49:14 (1786967354)
[ 5849.480683] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 07:50:22 (1786967422)
[ 5884.270868] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 07:50:57 (1786967457)
[ 5884.816961] Lustre: Mounted lustre-client - version 2.17.56_50_g3eb67fe
[ 6008.618359] Lustre: Unmounted lustre-client
[ 6017.272780] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 07:53:10 (1786967590)
[ 6018.412899] LustreError: 87667:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 6023.124499] LustreError: lustre-OST0000-osc-ffff95dbc2f6e800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6023.192796] LustreError: 87755:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff95dbc2f6e800: namespace resource [0x240000400:0x3e4d:0x0].0x0 (ffff95dbc8280400) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6023.256754] LustreError: 87755:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 3 previous similar messages
[ 6023.616578] LustreError: 87667:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout interrupted
[ 6023.620839] Lustre: 2358:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.103@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 6033.045788] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 07:53:26 (1786967606)
[ 6091.752372] Lustre: 88336:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786967611/real 1786967611]  req@ffff95dbc42dc700 x1873765154833920/t0(0) o101->lustre-MDT0000-mdc-ffff95dbc2f6e800@192.168.204.103@tcp:12/10 lens 328/344 e 0 to 1 dl 1786967666 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 6091.766971] Lustre: 88336:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 20 previous similar messages
[ 6099.523778] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 07:54:32 (1786967672)
[ 6101.370712] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 6103.383504] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 07:54:36 (1786967676)
[ 6169.042917] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6170.843717] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 07:55:44 (1786967744)
[ 6228.960241] Lustre: lustre-MDT0000-mdc-ffff95dbc2f6e800: Connection to lustre-MDT0000 (at 192.168.204.103@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6228.992674] Lustre: Skipped 15 previous similar messages
[ 6230.969570] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 07:56:44 (1786967804)
[ 6233.168078] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6235.138833] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 07:56:48 (1786967808)
[ 6236.744481] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6238.806391] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 07:56:52 (1786967812)
[ 6273.244549] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 07:57:26 (1786967846)
[ 6287.849641] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6317.024481] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 192.168.204.103@tcp) was lost; in progress operations using this service will fail
[ 6317.040714] LustreError: Skipped 2 previous similar messages
[ 6317.058165] Lustre: Evicted from MGS (at 192.168.204.103@tcp) after server handle changed from 0xeb1468f7631107 to 0xeb1468f7632933
[ 6317.063323] Lustre: Skipped 2 previous similar messages
[ 6317.067718] Lustre: MGC192.168.204.103@tcp: Connection restored to 192.168.204.103@tcp (at 192.168.204.103@tcp)
[ 6317.071924] Lustre: Skipped 15 previous similar messages
[ 6318.274800] Lustre: 68276:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.204.103@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6340.843222] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6342.306632] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6352.625950] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 07:58:46 (1786967926)
[ 6355.463401] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6357.638280] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 07:58:51 (1786967931)
[ 6373.376326] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 192.168.204.103@tcp) was lost; in progress operations using this service will fail
[ 6373.405220] Lustre: Evicted from MGS (at 192.168.204.103@tcp) after server handle changed from 0xeb1468f7632933 to 0xeb1468f7632dee
[ 6387.762606] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 07:59:20 (1786967960)
[ 6436.633891] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 08:00:09 (1786968009)
[ 6443.284908] LustreError: lustre-OST0000-osc-ffff95dbc2f6e800: operation ost_setattr to node 192.168.204.103@tcp failed: rc = -107
[ 6443.298379] LustreError: Skipped 1 previous similar message
[ 6494.392244] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6497.885038] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6582.240272] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 192.168.204.103@tcp) was lost; in progress operations using this service will fail
[ 6582.250895] LustreError: Skipped 1 previous similar message
[ 6592.518185] Lustre: Evicted from MGS (at 192.168.204.103@tcp) after server handle changed from 0xeb1468f7633175 to 0xeb1468f7633f44
[ 6592.527929] Lustre: Skipped 1 previous similar message
[ 6604.542523] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6606.155982] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6640.250478] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6641.568696] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6777.903905] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 08:05:51 (1786968351)
[ 6839.547111] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6842.568893] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 7092.982212] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 08:11:06 (1786968666)
[ 7247.865843] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 192.168.204.103@tcp) was lost; in progress operations using this service will fail
[ 7247.876403] LustreError: Skipped 1 previous similar message
[ 7247.881164] Lustre: lustre-MDT0000-mdc-ffff95dbc2f6e800: Connection to lustre-MDT0000 (at 192.168.204.103@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7247.905974] Lustre: Skipped 7 previous similar messages
[ 7247.908559] Lustre: Evicted from MGS (at 192.168.204.103@tcp) after server handle changed from 0xeb1468f7634350 to 0xeb1468f772e646
[ 7247.951083] Lustre: Skipped 1 previous similar message
[ 7247.984988] Lustre: MGC192.168.204.103@tcp: Connection restored to 192.168.204.103@tcp (at 192.168.204.103@tcp)
[ 7248.001757] Lustre: Skipped 9 previous similar messages
[ 7274.976176] Lustre: 2360:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786968812/real 1786968812]  req@ffff95dbc7b96d80 x1873765172529920/t0(0) o400->lustre-MDT0000-mdc-ffff95dbc2f6e800@192.168.204.103@tcp:12/10 lens 224/224 e 0 to 1 dl 1786968849 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7275.032781] Lustre: 2360:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 26 previous similar messages
[ 7281.844869] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 08:14:15 (1786968855)
[ 7283.466641] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7285.300913] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 08:14:18 (1786968858)
[ 7287.426953] LustreError: lustre-MDT0000-mdc-ffff95dbc2f6e800: operation ldlm_enqueue to node 192.168.204.103@tcp failed: rc = -107
[ 7287.442430] LustreError: Skipped 139 previous similar messages
[ 7287.461739] LustreError: lustre-MDT0000-mdc-ffff95dbc2f6e800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7295.320700] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 08:14:28 (1786968868)
[ 7462.192264] LustreError: lustre-OST0000-osc-ffff95dbc2f6e800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7464.144803] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 08:17:17 (1786969037)
[ 7508.021290] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 08:18:01 (1786969081)
[ 7509.876336] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7511.606359] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 08:18:05 (1786969085)
[ 7513.206577] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7515.013987] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 08:18:08 (1786969088)
[ 7548.075659] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 08:18:41 (1786969121)
[ 7589.358227] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 192.168.204.103@tcp) was lost; in progress operations using this service will fail
[ 7589.387950] Lustre: Evicted from MGS (at 192.168.204.103@tcp) after server handle changed from 0xeb1468f772e646 to 0xeb1468f7730ecc
[ 7589.398300] Lustre: Skipped 1 previous similar message
[ 7603.035632] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 08:19:36 (1786969176)
[ 7604.453458] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7606.400252] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 08:19:39 (1786969179)
[ 7607.848701] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7609.373297] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 08:19:43 (1786969183)
[ 7622.770885] Lustre: Unmounted lustre-client
[ 7624.708714] Lustre: Mounted lustre-client - version 2.17.56_50_g3eb67fe
[ 7627.386856] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7669.330339] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2