[ 1781.203557] LustreError: lustre-OST0000-osc-ffff8a71e044e800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1781.225975] Lustre: 2350:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.146@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1788.597696] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 01:26:49 (1789450009)
[ 1791.246790] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1793.449446] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 01:26:53 (1789450013)
[ 1795.164787] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1797.443421] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 01:26:57 (1789450017)
[ 1817.056266] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 192.168.204.146@tcp) was lost; in progress operations using this service will fail
[ 1827.243466] Lustre: Evicted from MGS (at 192.168.204.146@tcp) after server handle changed from 0x7803cd65546fb89f to 0x7803cd65546fd1b9
[ 1923.678237] LustreError: lustre-MDT0000-mdc-ffff8a71e044e800: operation mds_close to node 192.168.204.146@tcp failed: rc = -19
[ 1923.689160] LustreError: Skipped 4 previous similar messages
[ 1945.440252] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 192.168.204.146@tcp) was lost; in progress operations using this service will fail
[ 1945.506960] Lustre: Evicted from MGS (at 192.168.204.146@tcp) after server handle changed from 0x7803cd65546fd1b9 to 0x7803cd65547180fd
[ 1971.133834] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 01:29:51 (1789450191)
[ 1971.665714] Lustre: *** cfs_fail_loc=305, val=0***
[ 1971.668808] Lustre: Skipped 2 previous similar messages
[ 2010.081437] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 192.168.204.146@tcp) was lost; in progress operations using this service will fail
[ 2020.329959] Lustre: Evicted from MGS (at 192.168.204.146@tcp) after server handle changed from 0x7803cd65547180fd to 0x7803cd6554718445
[ 2020.356251] Lustre: MGC192.168.204.146@tcp: Connection restored to 192.168.204.146@tcp (at 192.168.204.146@tcp)
[ 2020.368452] Lustre: Skipped 14 previous similar messages
[ 2022.061592] Lustre: 15906:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.146@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2024.072512] Lustre: lustre-MDT0000-mdc-ffff8a71e044e800: Connection to lustre-MDT0000 (at 192.168.204.146@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2024.082056] Lustre: Skipped 12 previous similar messages
[ 2037.259159] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2038.706545] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2048.001544] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 01:31:08 (1789450268)
[ 2061.297298] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 192.168.204.146@tcp) was lost; in progress operations using this service will fail
[ 2061.328689] Lustre: Evicted from MGS (at 192.168.204.146@tcp) after server handle changed from 0x7803cd6554718445 to 0x7803cd6554718ace
[ 2071.681582] LustreError: lustre-MDT0000-mdc-ffff8a71e044e800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2088.959378] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 01:31:49 (1789450309)
[ 2103.399145] LustreError: lustre-OST0000-osc-ffff8a71e044e800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2125.990479] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 01:32:26 (1789450346)
[ 2140.628428] LustreError: lustre-MDT0000-mdc-ffff8a71e044e800: operation mds_reint to node 192.168.204.146@tcp failed: rc = -19
[ 2140.643813] LustreError: Skipped 1 previous similar message
[ 2159.584327] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 192.168.204.146@tcp) was lost; in progress operations using this service will fail
[ 2169.789088] Lustre: Evicted from MGS (at 192.168.204.146@tcp) after server handle changed from 0x7803cd6554718ace to 0x7803cd655471d38a
[ 2189.690959] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2192.178850] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2276.321479] Lustre: 2348:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789450482/real 1789450482]  req@ffff8a71d0225500 x1876372511571968/t0(0) o400->MGC192.168.204.146@tcp@192.168.204.146@tcp:26/25 lens 224/224 e 0 to 1 dl 1789450498 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2276.350878] Lustre: 2348:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 17 previous similar messages
[ 2276.358331] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 192.168.204.146@tcp) was lost; in progress operations using this service will fail
[ 2286.577913] Lustre: Evicted from MGS (at 192.168.204.146@tcp) after server handle changed from 0x7803cd655471d38a to 0x7803cd6554735d51
[ 2306.808533] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2309.825317] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2375.975134] LustreError: lustre-MDT0000-mdc-ffff8a71e044e800: operation ldlm_enqueue to node 192.168.204.146@tcp failed: rc = -19
[ 2375.986904] LustreError: Skipped 1 previous similar message
[ 2393.056397] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 192.168.204.146@tcp) was lost; in progress operations using this service will fail
[ 2403.255229] Lustre: Evicted from MGS (at 192.168.204.146@tcp) after server handle changed from 0x7803cd6554735d51 to 0x7803cd655474e202
[ 2421.821679] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2424.321482] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2453.874333] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 01:37:54 (1789450674)
[ 2479.072681] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 192.168.204.146@tcp) was lost; in progress operations using this service will fail
[ 2489.258572] Lustre: Evicted from MGS (at 192.168.204.146@tcp) after server handle changed from 0x7803cd655474e202 to 0x7803cd655475a9c1
[ 2496.543890] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2528.851664] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2573.608048] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2624.421252] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2624.621588] Lustre: lustre-MDT0000-mdc-ffff8a71e044e800: Connection restored to 192.168.204.146@tcp (at 192.168.204.146@tcp)
[ 2624.632071] Lustre: Skipped 16 previous similar messages
[ 2649.057499] LustreError: lustre-MDT0000-mdc-ffff8a71e044e800: operation ldlm_enqueue to node 192.168.204.146@tcp failed: rc = -19
[ 2649.067533] LustreError: Skipped 4 previous similar messages
[ 2649.075732] Lustre: lustre-MDT0000-mdc-ffff8a71e044e800: Connection to lustre-MDT0000 (at 192.168.204.146@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2649.095657] Lustre: Skipped 8 previous similar messages
[ 2666.208687] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 192.168.204.146@tcp) was lost; in progress operations using this service will fail
[ 2666.222917] LustreError: Skipped 3 previous similar messages
[ 2676.394278] Lustre: Evicted from MGS (at 192.168.204.146@tcp) after server handle changed from 0x7803cd6554762f46 to 0x7803cd6554769487
[ 2676.408847] Lustre: Skipped 3 previous similar messages
[ 2686.416923] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2743.097651] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2829.883432] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 01:44:10 (1789451050)
[ 2887.714108] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 2890.276945] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3175.691966] LustreError: lustre-OST0000-osc-ffff8a71e044e800: operation ldlm_enqueue to node 192.168.204.146@tcp failed: rc = -19
[ 3175.704155] LustreError: Skipped 4 previous similar messages
[ 3217.560256] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3220.071100] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3507.351186] Lustre: lustre-OST0000-osc-ffff8a71e044e800: Connection to lustre-OST0000 (at 192.168.204.146@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3507.383921] Lustre: Skipped 4 previous similar messages
[ 3533.230147] Lustre: lustre-OST0000-osc-ffff8a71e044e800: Connection restored to 192.168.204.146@tcp (at 192.168.204.146@tcp)
[ 3533.236368] Lustre: Skipped 8 previous similar messages
[ 3551.350436] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3553.617396] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3802.361401] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 02:00:23 (1789452023)
[ 3818.976192] Lustre: 50552:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789452025/real 1789452025]  req@ffff8a71c7fcb800 x1876372531292032/t0(0) o101->lustre-MDT0000-mdc-ffff8a71e044e800@192.168.204.146@tcp:12/10 lens 576/1152 e 0 to 1 dl 1789452041 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3818.999429] Lustre: 50552:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
[ 3827.459311] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 02:00:48 (1789452048)
[ 3854.924758] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 02:01:15 (1789452075)
[ 3882.121165] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 02:01:42 (1789452102)
[ 3882.711755] Lustre: Mounted lustre-client - version 2.17.58_40_gc6c2479
[ 3914.720681] Lustre: 2350:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789452120/real 1789452120]  req@ffff8a71e0179f80 x1876372531310976/t0(0) o400->MGC192.168.204.146@tcp@192.168.204.146@tcp:26/25 lens 224/224 e 0 to 1 dl 1789452136 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3914.752192] Lustre: 2350:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 3914.759929] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 192.168.204.146@tcp) was lost; in progress operations using this service will fail
[ 3914.788654] LustreError: Skipped 2 previous similar messages
[ 3923.961196] Lustre: Evicted from MGS (at 192.168.204.146@tcp) after server handle changed from 0x7803cd655477cbac to 0x7803cd65548a7a42
[ 3923.971702] Lustre: Skipped 2 previous similar messages
[ 3925.685462] Lustre: 15906:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.146@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 3945.040864] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 3946.893357] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3949.214600] Lustre: Unmounted lustre-client
[ 3955.955319] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 02:02:56 (1789452176)
[ 4077.536172] Lustre: 2350:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789452283/real 1789452283]  req@ffff8a71e01c0a80 x1876372531345536/t0(0) o4->lustre-OST0000-osc-ffff8a71e044e800@192.168.204.146@tcp:6/4 lens 488/448 e 0 to 1 dl 1789452299 ref 2 fl Rpc:XQr/602/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4077.565033] Lustre: 2350:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 72 previous similar messages
[ 4109.280278] Lustre: lustre-OST0000-osc-ffff8a71e044e800: Connection to lustre-OST0000 (at 192.168.204.146@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4109.304881] Lustre: Skipped 13 previous similar messages
[ 4141.059866] Lustre: lustre-OST0000-osc-ffff8a71e044e800: Connection restored to 192.168.204.146@tcp (at 192.168.204.146@tcp)
[ 4141.065190] Lustre: Skipped 16 previous similar messages
[ 4253.783710] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 02:07:54 (1789452474)
[ 4306.974343] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 02:08:47 (1789452527)
[ 4310.324386] Lustre: Unmounted lustre-client
[ 4335.775359] Lustre: Mounted lustre-client - version 2.17.58_40_gc6c2479
[ 4342.799794] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 02:09:23 (1789452563)
[ 4343.014397] LustreError: 56494:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4344.064081] LustreError: 56494:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4344.250775] Lustre: *** cfs_fail_loc=305, val=0***
[ 4367.481248] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 02:09:47 (1789452587)
[ 4367.995726] Lustre: Mounted lustre-client - version 2.17.58_40_gc6c2479
[ 4370.148551] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4380.535871] Lustre: Unmounted lustre-client
[ 4388.673745] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 02:10:09 (1789452609)
[ 4494.922609] LustreError: lustre-MDT0000-mdc-ffff8a71d02ca800: operation mds_reint to node 192.168.204.146@tcp failed: rc = -19
[ 4494.932905] LustreError: Skipped 1 previous similar message
[ 4516.842653] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 192.168.204.146@tcp) was lost; in progress operations using this service will fail
[ 4516.864618] Lustre: Evicted from MGS (at 192.168.204.146@tcp) after server handle changed from 0x7803cd65548a9bba to 0x7803cd65548d5035
[ 4518.703174] Lustre: 55892:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.146@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 4708.371829] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 02:15:28 (1789452928)
[ 4713.357733] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4730.866251] Lustre: lustre-MDT0000-mdc-ffff8a71d02ca800: Connection to lustre-MDT0000 (at 192.168.204.146@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4730.881977] Lustre: Skipped 11 previous similar messages
[ 4730.885701] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 192.168.204.146@tcp) was lost; in progress operations using this service will fail
[ 4730.917693] Lustre: Evicted from MGS (at 192.168.204.146@tcp) after server handle changed from 0x7803cd65548d5035 to 0x7803cd65549219eb
[ 4730.995380] LustreError: lustre-MDT0000-mdc-ffff8a71d02ca800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4745.620243] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 02:16:06 (1789452966)
[ 4746.091674] Lustre: Mounted lustre-client - version 2.17.58_40_gc6c2479
[ 4753.543284] LustreError: lustre-OST0000-osc-ffff8a71d02ca800: operation ldlm_enqueue to node 192.168.204.146@tcp failed: rc = -107
[ 4753.572472] LustreError: lustre-OST0000-osc-ffff8a71d02ca800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4763.616381] Lustre: 2350:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789452969/real 1789452969]  req@ffff8a71dabcca80 x1876372535996416/t0(0) o103->lustre-OST0000-osc-ffff8a71da9f8800@192.168.204.146@tcp:17/18 lens 328/224 e 0 to 1 dl 1789452985 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295
[ 4763.686896] Lustre: 2350:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 83 previous similar messages
[ 4763.767889] Lustre: lustre-OST0000-osc-ffff8a71da9f8800: Connection restored to 192.168.204.146@tcp (at 192.168.204.146@tcp)
[ 4763.785724] Lustre: Skipped 12 previous similar messages
[ 4765.640726] Lustre: Unmounted lustre-client
[ 4773.558870] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 02:16:34 (1789452994)
[ 4779.891214] LustreError: lustre-MDT0000-mdc-ffff8a71d02ca800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4779.915852] LustreError: 60891:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4788.189405] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 02:16:48 (1789453008)
[ 4788.507586] LustreError: 61595:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4792.500227] LustreError: lustre-MDT0000-mdc-ffff8a71d02ca800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4792.522375] LustreError: 61612:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4792.536375] LustreError: 61595:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=977
[ 4792.546189] LustreError: 61595:0:(import.c:762:ptlrpc_connect_import_locked()) already connecting
[ 4792.594299] LustreError: 61617:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4793.666936] LustreError: 61623:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4793.683974] LustreError: 61623:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4796.046474] LustreError: 61646:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4796.067188] LustreError: 61646:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4800.728195] LustreError: 61691:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4800.737075] LustreError: 61691:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4811.044639] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 02:17:11 (1789453031)
[ 4852.683345] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4854.442252] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4865.741823] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 02:18:06 (1789453086)
[ 4904.908487] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4906.301506] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4916.231851] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 02:18:56 (1789453136)
[ 4983.707776] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4985.709958] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4996.113775] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 02:20:15 (1789453215)
[ 5038.842861] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5040.559803] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5046.408110] Lustre: Unmounted lustre-client
[ 5102.848992] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5105.881836] Lustre: Mounted lustre-client - version 2.17.58_40_gc6c2479
[ 5115.120487] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 02:22:15 (1789453335)
[ 5118.254177] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5120.886396] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 02:22:21 (1789453341)
[ 5156.249447] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 02:22:57 (1789453377)
[ 5157.666551] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5159.451562] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 02:23:00 (1789453380)
[ 5161.222795] Lustre: *** cfs_fail_loc=805, val=0***
[ 5161.319726] Lustre: Mounted lustre-client - version 2.17.58_40_gc6c2479
[ 5167.056439] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5187.041884] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 192.168.204.146@tcp) was lost; in progress operations using this service will fail
[ 5197.307183] Lustre: Evicted from MGS (at 192.168.204.146@tcp) after server handle changed from 0x7803cd65549223df to 0x7803cd6554922789
[ 5197.320140] LustreError: 2347:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8a71d8f4b000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff8a71c5afc600) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5199.992040] Lustre: Unmounted lustre-client
[ 5211.515291] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 02:23:51 (1789453431)
[ 5238.325467] LustreError: 2347:0:(client.c:3449:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff8a71e00e5f80 x1876372536078976/t90194313220(90194313220) o101->lustre-MDT0000-mdc-ffff8a71e0823000@192.168.204.146@tcp:12/10 lens 648/608 e 0 to 0 dl 1789453476 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5251.626293] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5253.730167] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5263.292919] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 02:24:43 (1789453483)
[ 5264.942663] LustreError: lustre-OST0000-osc-ffff8a71e0823000: operation ost_write to node 192.168.204.146@tcp failed: rc = -107
[ 5264.960293] LustreError: Skipped 2 previous similar messages
[ 5264.978299] LustreError: lustre-OST0000-osc-ffff8a71e0823000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5264.996482] Lustre: 2349:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.146@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5265.029032] LustreError: 72801:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff8a71e0823000: namespace resource [0x240000400:0x3ae2:0x0].0x0 (ffff8a71c5b45000) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5265.052322] LustreError: 72801:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5273.651736] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 02:24:54 (1789453494)
[ 5275.423636] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5277.266734] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 02:24:57 (1789453497)
[ 5278.895426] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5280.885781] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 02:25:01 (1789453501)
[ 5282.360143] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5284.142461] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 02:25:04 (1789453504)
[ 5285.727436] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5287.585953] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 02:25:08 (1789453508)
[ 5289.033536] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5290.642356] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 02:25:11 (1789453511)
[ 5292.191846] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5294.062144] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 02:25:14 (1789453514)
[ 5295.867699] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5297.547451] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 02:25:18 (1789453518)
[ 5299.190476] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5301.026880] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 02:25:21 (1789453521)
[ 5302.554635] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5304.263982] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 02:25:24 (1789453524)
[ 5305.541422] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5306.996875] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 02:25:27 (1789453527)
[ 5308.360207] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5310.348904] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 02:25:31 (1789453531)
[ 5312.790033] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5315.069970] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 02:25:35 (1789453535)
[ 5336.036558] Lustre: lustre-MDT0000-mdc-ffff8a71e0823000: Connection to lustre-MDT0000 (at 192.168.204.146@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5336.056047] Lustre: Skipped 14 previous similar messages
[ 5352.336902] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 02:26:12 (1789453572)
[ 5384.231631] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 02:26:44 (1789453604)
[ 5385.020395] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5395.991828] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 02:26:56 (1789453616)
[ 5396.332839] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5408.175437] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 02:27:08 (1789453628)
[ 5408.737237] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5417.993721] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 02:27:18 (1789453638)
[ 5418.297917] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5427.188418] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 02:27:27 (1789453647)
[ 5427.643848] Lustre: *** cfs_fail_loc=510, val=3***
[ 5437.088693] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 02:27:37 (1789453657)
[ 5437.404359] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5496.288410] Lustre: 2351:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789453663/real 1789453663]  req@ffff8a71dabcfb80 x1876372536121984/t0(0) o3->lustre-OST0000-osc-ffff8a71e0823000@192.168.204.146@tcp:6/4 lens 488/4536 e 0 to 1 dl 1789453718 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 5496.328594] Lustre: 2351:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 16 previous similar messages
[ 5504.166792] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 02:28:44 (1789453724)
[ 5504.692918] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5571.819293] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 02:29:52 (1789453792)
[ 5572.374688] LustreError: 83017:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5575.397183] LustreError: lustre-MDT0000-mdc-ffff8a71e0823000: operation ldlm_enqueue to node 192.168.204.146@tcp failed: rc = -107
[ 5575.416050] LustreError: lustre-MDT0000-mdc-ffff8a71e0823000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5575.448231] LustreError: 83031:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8a71e0823000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8a71c5b45a00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5575.474071] Lustre: lustre-MDT0000-mdc-ffff8a71e0823000: Connection restored to 192.168.204.146@tcp (at 192.168.204.146@tcp)
[ 5575.481239] Lustre: Skipped 14 previous similar messages
[ 5576.457115] LustreError: 83017:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5578.556617] LustreError: 83047:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5581.574227] LustreError: lustre-MDT0000-mdc-ffff8a71e0823000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5581.603969] LustreError: 83062:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8a71e0823000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8a71c5baff00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5581.648463] LustreError: 83062:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5581.662349] LustreError: 83067:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5581.685544] LustreError: 83067:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5582.648209] LustreError: 83047:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5589.601665] LustreError: lustre-MDT0000-mdc-ffff8a71e0823000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5589.616374] LustreError: 83090:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8a71e0823000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff8a71c5afcf00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5589.629389] LustreError: 83090:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5590.081273] LustreError: 83103:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5593.173569] LustreError: lustre-MDT0000-mdc-ffff8a71e0823000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5593.243985] LustreError: 83123:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5594.144209] LustreError: 83103:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5594.154488] LustreError: 83103:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8a71e0823000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5595.495209] LustreError: 83141:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5595.505839] LustreError: 83141:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 12 previous similar messages
[ 5600.281539] LustreError: 83179:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5603.274846] LustreError: lustre-MDT0000-mdc-ffff8a71e0823000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5603.293503] LustreError: 83193:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8a71e0823000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8a71c5bafc00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5604.368141] LustreError: 83179:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5611.471288] LustreError: lustre-MDT0000-mdc-ffff8a71e0823000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5611.972363] LustreError: 83234:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5614.929955] LustreError: lustre-MDT0000-mdc-ffff8a71e0823000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5616.074368] LustreError: 83234:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5619.096495] LustreError: lustre-MDT0000-mdc-ffff8a71e0823000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5619.181979] LustreError: 83282:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5619.205017] LustreError: 83282:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5620.284252] LustreError: 83262:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8a71e0823000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5628.425803] LustreError: 83334:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5628.436885] LustreError: 83334:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5632.158244] LustreError: lustre-MDT0000-mdc-ffff8a71e0823000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5632.257775] LustreError: 83356:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5632.268114] LustreError: 83356:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5632.512248] LustreError: 83334:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5632.521356] LustreError: 83334:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5632.548288] LustreError: 83334:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8a71e0823000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5643.934585] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 02:31:04 (1789453864)
[ 5664.057858] LustreError: 81053:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5664.061888] LustreError: 81053:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5668.128163] LustreError: 81053:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5668.140905] LustreError: 81053:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5675.118614] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 02:31:35 (1789453895)
[ 5727.253885] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 02:32:27 (1789453947)
[ 5794.835343] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 02:33:35 (1789454015)
[ 5830.127455] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 02:34:11 (1789454051)
[ 5830.579750] Lustre: Mounted lustre-client - version 2.17.58_40_gc6c2479
[ 5955.399614] Lustre: Unmounted lustre-client
[ 5964.592672] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 02:36:25 (1789454185)
[ 5965.578577] LustreError: 87662:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5968.783656] Lustre: lustre-OST0000-osc-ffff8a71e0823000: Connection to lustre-OST0000 (at 192.168.204.146@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5968.801436] Lustre: Skipped 14 previous similar messages
[ 5968.826694] LustreError: lustre-OST0000-osc-ffff8a71e0823000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5968.846359] LustreError: 87744:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff8a71e0823000: namespace resource [0x240000400:0x3b0d:0x0].0x0 (ffff8a71c5b45500) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5968.865044] LustreError: 87744:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 3 previous similar messages
[ 5969.248155] LustreError: 87662:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5969.254695] Lustre: 2350:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.146@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5977.361374] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 02:36:37 (1789454197)
[ 6045.357644] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 02:37:45 (1789454265)
[ 6046.891872] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 6048.654577] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 02:37:49 (1789454269)
[ 6106.080172] Lustre: 89327:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789454272/real 1789454272]  req@ffff8a71c7fcaa00 x1876372536266880/t0(0) o101->lustre-MDT0000-mdc-ffff8a71e0823000@192.168.204.146@tcp:12/10 lens 648/66320 e 0 to 1 dl 1789454328 ref 2 fl Rpc:XPQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 6106.111651] Lustre: 89327:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
[ 6113.474763] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6115.248067] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 02:38:55 (1789454335)
[ 6175.557302] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 02:39:56 (1789454396)
[ 6178.128404] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6180.584897] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 02:40:00 (1789454400)
[ 6181.986986] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6184.008614] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 02:40:04 (1789454404)
[ 6222.404340] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 02:40:42 (1789454442)
[ 6235.543261] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6261.728299] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 192.168.204.146@tcp) was lost; in progress operations using this service will fail
[ 6261.733876] LustreError: Skipped 2 previous similar messages
[ 6266.975214] Lustre: lustre-MDT0000-mdc-ffff8a71e0823000: Connection restored to 192.168.204.146@tcp (at 192.168.204.146@tcp)
[ 6266.985667] Lustre: Skipped 14 previous similar messages
[ 6271.044184] Lustre: Evicted from MGS (at 192.168.204.146@tcp) after server handle changed from 0x7803cd655492322c to 0x7803cd6554924cd5
[ 6271.055376] Lustre: Skipped 2 previous similar messages
[ 6281.456303] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6284.084548] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6298.156773] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 02:41:58 (1789454518)
[ 6299.966388] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6301.913967] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 02:42:02 (1789454522)
[ 6317.076612] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 192.168.204.146@tcp) was lost; in progress operations using this service will fail
[ 6317.100114] Lustre: Evicted from MGS (at 192.168.204.146@tcp) after server handle changed from 0x7803cd6554924cd5 to 0x7803cd6554924ef0
[ 6329.971689] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 02:42:30 (1789454550)
[ 6359.667839] Lustre: 68258:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.146@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6382.371447] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 02:43:22 (1789454602)
[ 6388.042104] LustreError: lustre-OST0000-osc-ffff8a71e0823000: operation ost_setattr to node 192.168.204.146@tcp failed: rc = -19
[ 6388.056091] LustreError: Skipped 10 previous similar messages
[ 6446.081104] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6449.722161] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6536.160852] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 192.168.204.146@tcp) was lost; in progress operations using this service will fail
[ 6536.173089] LustreError: Skipped 1 previous similar message
[ 6546.420497] Lustre: Evicted from MGS (at 192.168.204.146@tcp) after server handle changed from 0x7803cd65549250fd to 0x7803cd6554925f51
[ 6546.434145] Lustre: Skipped 1 previous similar message
[ 6548.080809] Lustre: 68258:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.146@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6565.748732] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6567.795261] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6593.645770] Lustre: 68258:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.146@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6599.862212] Lustre: lustre-MDT0000-mdc-ffff8a71e0823000: Connection to lustre-MDT0000 (at 192.168.204.146@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6599.879463] Lustre: Skipped 8 previous similar messages
[ 6615.622840] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6617.590136] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6766.031179] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 02:49:46 (1789454986)
[ 6828.913694] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6832.668212] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 7092.979906] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 02:55:13 (1789455313)
[ 7263.237726] Lustre: lustre-MDT0000-mdc-ffff8a71e0823000: Connection to lustre-MDT0000 (at 192.168.204.146@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7263.285375] Lustre: Skipped 1 previous similar message
[ 7283.705132] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 192.168.204.146@tcp) was lost; in progress operations using this service will fail
[ 7283.729217] LustreError: Skipped 1 previous similar message
[ 7283.771697] Lustre: Evicted from MGS (at 192.168.204.146@tcp) after server handle changed from 0x7803cd6554926284 to 0x7803cd6554a2068b
[ 7283.778204] Lustre: Skipped 1 previous similar message
[ 7283.784454] Lustre: MGC192.168.204.146@tcp: Connection restored to 192.168.204.146@tcp (at 192.168.204.146@tcp)
[ 7283.792565] Lustre: Skipped 9 previous similar messages
[ 7316.560202] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 02:58:57 (1789455537)
[ 7318.461628] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7320.684183] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 02:59:00 (1789455540)
[ 7322.986478] LustreError: lustre-MDT0000-mdc-ffff8a71e0823000: operation ldlm_enqueue to node 192.168.204.146@tcp failed: rc = -107
[ 7322.997509] LustreError: Skipped 250 previous similar messages
[ 7323.010346] LustreError: lustre-MDT0000-mdc-ffff8a71e0823000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7332.156252] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 02:59:12 (1789455552)
[ 7499.005548] LustreError: lustre-OST0000-osc-ffff8a71e0823000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7501.115661] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 03:02:01 (1789455721)
[ 7526.880181] Lustre: 2350:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789455728/real 1789455728]  req@ffff8a71dabee680 x1876372553997440/t0(0) o4->lustre-OST0000-osc-ffff8a71e0823000@192.168.204.146@tcp:6/4 lens 4584/448 e 0 to 1 dl 1789455748 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7526.933074] Lustre: 2350:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 27 previous similar messages
[ 7545.751574] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 03:02:46 (1789455766)
[ 7547.687657] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7549.664481] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 03:02:50 (1789455770)
[ 7551.919918] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7554.614237] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 03:02:54 (1789455774)
[ 7592.430341] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 03:03:32 (1789455812)
[ 7632.370991] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 192.168.204.146@tcp) was lost; in progress operations using this service will fail
[ 7632.414714] Lustre: Evicted from MGS (at 192.168.204.146@tcp) after server handle changed from 0x7803cd6554a2068b to 0x7803cd6554a22d19
[ 7632.432162] Lustre: Skipped 1 previous similar message
[ 7649.922708] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 03:04:29 (1789455869)
[ 7652.625881] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7655.935740] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 03:04:35 (1789455875)
[ 7658.867548] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7658.981730] Lustre: 2349:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789455844/real 1789455844]  req@ffff8a71c86a8000 x1876372554153088/t0(0) o400->lustre-MDT0000-mdc-ffff8a71e0823000@192.168.204.146@tcp:12/10 lens 224/224 e 0 to 1 dl 1789455880 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7661.228936] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 03:04:41 (1789455881)
[ 7672.301903] Lustre: Unmounted lustre-client
[ 7674.923964] Lustre: Mounted lustre-client - version 2.17.58_40_gc6c2479
[ 7678.490048] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7700.457874] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 192.168.204.146@tcp) was lost; in progress operations using this service will fail
[ 7701.941215] Lustre: 106819:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.146@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 7718.225736] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2