[ 1919.768881] LustreError: lustre-OST0000-osc-ffff9c7450f1a800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1919.788524] Lustre: 2359:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.133@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1925.985626] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 01:49:32 (1788587372)
[ 1927.780498] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1929.324199] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 01:49:35 (1788587375)
[ 1930.926721] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1932.570634] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 01:49:38 (1788587378)
[ 1936.314478] LustreError: lustre-MDT0000-mdc-ffff9c7450f1a800: operation ldlm_enqueue to node 192.168.203.133@tcp failed: rc = -19
[ 1936.320654] LustreError: Skipped 1 previous similar message
[ 1955.872121] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 192.168.203.133@tcp) was lost; in progress operations using this service will fail
[ 1966.068060] Lustre: Evicted from MGS (at 192.168.203.133@tcp) after server handle changed from 0x70985678e70bf6ab to 0x70985678e70c0e2f
[ 2070.285518] LustreError: lustre-MDT0000-mdc-ffff9c7450f1a800: operation ldlm_enqueue to node 192.168.203.133@tcp failed: rc = -19
[ 2087.839612] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 192.168.203.133@tcp) was lost; in progress operations using this service will fail
[ 2098.022256] Lustre: Evicted from MGS (at 192.168.203.133@tcp) after server handle changed from 0x70985678e70c0e2f to 0x70985678e70e1e8c
[ 2114.541863] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 01:52:40 (1788587560)
[ 2115.059187] Lustre: *** cfs_fail_loc=305, val=0***
[ 2115.062020] Lustre: Skipped 2 previous similar messages
[ 2154.464327] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 192.168.203.133@tcp) was lost; in progress operations using this service will fail
[ 2154.491118] Lustre: Evicted from MGS (at 192.168.203.133@tcp) after server handle changed from 0x70985678e70e1e8c to 0x70985678e70e2275
[ 2168.993503] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2171.016449] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2179.638870] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 01:53:45 (1788587625)
[ 2195.436490] Lustre: lustre-MDT0000-mdc-ffff9c7450f1a800: Connection to lustre-MDT0000 (at 192.168.203.133@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2195.467053] Lustre: Skipped 12 previous similar messages
[ 2195.470742] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 192.168.203.133@tcp) was lost; in progress operations using this service will fail
[ 2195.518465] Lustre: Evicted from MGS (at 192.168.203.133@tcp) after server handle changed from 0x70985678e70e2275 to 0x70985678e70e276f
[ 2195.529303] Lustre: MGC192.168.203.133@tcp: Connection restored to 192.168.203.133@tcp (at 192.168.203.133@tcp)
[ 2195.541137] Lustre: Skipped 15 previous similar messages
[ 2195.551954] LustreError: lustre-MDT0000-mdc-ffff9c7450f1a800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2216.123941] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 01:54:22 (1788587662)
[ 2230.533369] LustreError: lustre-OST0000-osc-ffff9c7450f1a800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2248.276533] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 01:54:54 (1788587694)
[ 2262.605697] LustreError: lustre-MDT0000-mdc-ffff9c7450f1a800: operation mds_reint to node 192.168.203.133@tcp failed: rc = -19
[ 2283.551985] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 192.168.203.133@tcp) was lost; in progress operations using this service will fail
[ 2283.589496] Lustre: Evicted from MGS (at 192.168.203.133@tcp) after server handle changed from 0x70985678e70e276f to 0x70985678e70e7326
[ 2285.506469] Lustre: 15921:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.133@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2309.664995] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2311.929581] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2394.911295] Lustre: 2357:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788587826/real 1788587826]  req@ffff9c7449400a80 x1875467849968256/t0(0) o400->MGC192.168.203.133@tcp@192.168.203.133@tcp:26/25 lens 224/224 e 0 to 1 dl 1788587842 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2394.936774] Lustre: 2357:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 12 previous similar messages
[ 2394.942508] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 192.168.203.133@tcp) was lost; in progress operations using this service will fail
[ 2405.163850] Lustre: Evicted from MGS (at 192.168.203.133@tcp) after server handle changed from 0x70985678e70e7326 to 0x70985678e7102b1c
[ 2422.899953] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2424.980152] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2491.221823] LustreError: lustre-MDT0000-mdc-ffff9c7450f1a800: operation ldlm_enqueue to node 192.168.203.133@tcp failed: rc = -19
[ 2491.229379] LustreError: Skipped 1 previous similar message
[ 2510.836736] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 192.168.203.133@tcp) was lost; in progress operations using this service will fail
[ 2510.868997] Lustre: Evicted from MGS (at 192.168.203.133@tcp) after server handle changed from 0x70985678e7102b1c to 0x70985678e711bed7
[ 2526.366750] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2528.521934] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2556.884863] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 02:00:03 (1788588003)
[ 2578.399306] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 192.168.203.133@tcp) was lost; in progress operations using this service will fail
[ 2586.813471] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2588.605126] Lustre: Evicted from MGS (at 192.168.203.133@tcp) after server handle changed from 0x70985678e711bed7 to 0x70985678e712c813
[ 2606.562203] Lustre: 15921:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.133@tcp: IR log lustre-cliir failed, not fatal: rc = -5
[ 2626.086284] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2662.407848] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2702.773324] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2745.312040] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 192.168.203.133@tcp) was lost; in progress operations using this service will fail
[ 2745.329369] LustreError: Skipped 3 previous similar messages
[ 2745.368776] Lustre: Evicted from MGS (at 192.168.203.133@tcp) after server handle changed from 0x70985678e7131ec8 to 0x70985678e7136bac
[ 2745.380865] Lustre: Skipped 3 previous similar messages
[ 2753.127282] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2781.214900] LustreError: lustre-MDT0000-mdc-ffff9c7450f1a800: operation mds_reint to node 192.168.203.133@tcp failed: rc = -19
[ 2781.233318] LustreError: Skipped 5 previous similar messages
[ 2801.270330] Lustre: MGC192.168.203.133@tcp: Connection restored to 192.168.203.133@tcp (at 192.168.203.133@tcp)
[ 2801.274933] Lustre: Skipped 17 previous similar messages
[ 2808.080435] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2841.460455] Lustre: lustre-MDT0000-mdc-ffff9c7450f1a800: Connection to lustre-MDT0000 (at 192.168.203.133@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2841.479984] Lustre: Skipped 9 previous similar messages
[ 2904.402211] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 02:05:50 (1788588350)
[ 2958.575609] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 2960.859262] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3302.874614] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3306.190489] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3583.183346] LustreError: lustre-OST0000-osc-ffff9c7450f1a800: operation ldlm_enqueue to node 192.168.203.133@tcp failed: rc = -19
[ 3583.212536] LustreError: Skipped 12 previous similar messages
[ 3583.229957] Lustre: lustre-OST0000-osc-ffff9c7450f1a800: Connection to lustre-OST0000 (at 192.168.203.133@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3583.272861] Lustre: Skipped 2 previous similar messages
[ 3605.764955] Lustre: lustre-OST0000-osc-ffff9c7450f1a800: Connection restored to 192.168.203.133@tcp (at 192.168.203.133@tcp)
[ 3605.779435] Lustre: Skipped 5 previous similar messages
[ 3625.438558] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3627.680906] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3880.187694] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 02:22:06 (1788589326)
[ 3898.337933] Lustre: 50581:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788589329/real 1788589329]  req@ffff9c74526b3800 x1875467870018944/t0(0) o101->lustre-MDT0000-mdc-ffff9c7450f1a800@192.168.203.133@tcp:12/10 lens 576/1152 e 0 to 1 dl 1788589345 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3898.385347] Lustre: 50581:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
[ 3905.371981] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 02:22:31 (1788589351)
[ 3931.144602] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 02:22:57 (1788589377)
[ 3959.788810] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 02:23:25 (1788589405)
[ 3960.428915] Lustre: Mounted lustre-client - version 2.17.58_3_g357faf0
[ 3992.545137] Lustre: 2359:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788589423/real 1788589423]  req@ffff9c745260c380 x1875467870038144/t0(0) o400->MGC192.168.203.133@tcp@192.168.203.133@tcp:26/25 lens 224/224 e 0 to 1 dl 1788589439 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3992.585266] Lustre: 2359:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 3992.605995] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 192.168.203.133@tcp) was lost; in progress operations using this service will fail
[ 3992.621228] LustreError: Skipped 2 previous similar messages
[ 4002.805714] Lustre: Evicted from MGS (at 192.168.203.133@tcp) after server handle changed from 0x70985678e714620a to 0x70985678e727bc0b
[ 4002.821556] Lustre: Skipped 2 previous similar messages
[ 4010.294448] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4012.071428] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4014.286478] Lustre: Unmounted lustre-client
[ 4021.192341] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 02:24:27 (1788589467)
[ 4158.431885] Lustre: 2357:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788589589/real 1788589589]  req@ffff9c74526b0000 x1875467870072448/t0(0) o4->lustre-OST0000-osc-ffff9c7450f1a800@192.168.203.133@tcp:6/4 lens 488/448 e 0 to 1 dl 1788589605 ref 2 fl Rpc:XQr/602/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4158.459660] Lustre: 2357:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 73 previous similar messages
[ 4190.175300] Lustre: lustre-OST0000-osc-ffff9c7450f1a800: Connection to lustre-OST0000 (at 192.168.203.133@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4190.195288] Lustre: Skipped 14 previous similar messages
[ 4206.589365] Lustre: lustre-OST0000-osc-ffff9c7450f1a800: Connection restored to 192.168.203.133@tcp (at 192.168.203.133@tcp)
[ 4206.609151] Lustre: Skipped 16 previous similar messages
[ 4317.586956] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 02:29:23 (1788589763)
[ 4368.277609] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 02:30:14 (1788589814)
[ 4372.174405] Lustre: Unmounted lustre-client
[ 4395.798515] Lustre: Mounted lustre-client - version 2.17.58_3_g357faf0
[ 4403.346756] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 02:30:49 (1788589849)
[ 4403.600374] LustreError: 56516:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4404.648762] LustreError: 56516:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4404.836853] Lustre: *** cfs_fail_loc=305, val=0***
[ 4430.532039] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 02:31:16 (1788589876)
[ 4431.145319] Lustre: Mounted lustre-client - version 2.17.58_3_g357faf0
[ 4434.173792] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4444.528815] Lustre: Unmounted lustre-client
[ 4452.967291] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 02:31:38 (1788589898)
[ 4554.573881] LustreError: lustre-MDT0000-mdc-ffff9c744822e800: operation mds_reint to node 192.168.203.133@tcp failed: rc = -19
[ 4574.687219] Lustre: 2358:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788590006/real 1788590006]  req@ffff9c7446f50380 x1875467872738944/t0(0) o400->MGC192.168.203.133@tcp@192.168.203.133@tcp:26/25 lens 224/224 e 0 to 1 dl 1788590022 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4574.725136] Lustre: 2358:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 74 previous similar messages
[ 4574.739759] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 192.168.203.133@tcp) was lost; in progress operations using this service will fail
[ 4584.931651] Lustre: Evicted from MGS (at 192.168.203.133@tcp) after server handle changed from 0x70985678e727d437 to 0x70985678e72a8459
[ 4766.889611] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 02:36:52 (1788590212)
[ 4772.204161] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4788.718841] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 192.168.203.133@tcp) was lost; in progress operations using this service will fail
[ 4788.740831] Lustre: Evicted from MGS (at 192.168.203.133@tcp) after server handle changed from 0x70985678e72a8459 to 0x70985678e72f57f5
[ 4788.787773] LustreError: lustre-MDT0000-mdc-ffff9c744822e800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4802.600873] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 02:37:28 (1788590248)
[ 4803.210487] Lustre: Mounted lustre-client - version 2.17.58_3_g357faf0
[ 4810.625458] Lustre: lustre-OST0000-osc-ffff9c744822e800: Connection to lustre-OST0000 (at 192.168.203.133@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4810.643604] Lustre: Skipped 11 previous similar messages
[ 4810.663898] LustreError: lustre-OST0000-osc-ffff9c744822e800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4821.002832] Lustre: lustre-OST0000-osc-ffff9c7452b62000: Connection restored to 192.168.203.133@tcp (at 192.168.203.133@tcp)
[ 4821.019737] Lustre: Skipped 12 previous similar messages
[ 4823.139402] Lustre: Unmounted lustre-client
[ 4829.801285] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 02:37:55 (1788590275)
[ 4836.971412] LustreError: lustre-MDT0000-mdc-ffff9c744822e800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4836.998360] LustreError: 60913:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4845.021294] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 02:38:10 (1788590290)
[ 4845.240574] LustreError: 61612:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4849.556131] LustreError: lustre-MDT0000-mdc-ffff9c744822e800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4849.575934] LustreError: 61634:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4849.588594] LustreError: 61612:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=662
[ 4849.599860] LustreError: 61612:0:(import.c:761:ptlrpc_connect_import_locked()) already connecting
[ 4849.666248] LustreError: 61639:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4850.763859] LustreError: 61645:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4850.777468] LustreError: 61645:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4853.134409] LustreError: 61668:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4853.151678] LustreError: 61668:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4857.776152] LustreError: 61713:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4857.796946] LustreError: 61713:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4868.413432] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 02:38:34 (1788590314)
[ 4907.403818] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4909.296502] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4921.996675] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 02:39:27 (1788590367)
[ 4959.388743] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4961.185296] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4971.178213] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 02:40:17 (1788590417)
[ 5033.742838] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5035.595244] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5044.380756] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 02:41:30 (1788590490)
[ 5085.403781] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5086.920162] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5093.019657] Lustre: Unmounted lustre-client
[ 5144.600850] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5147.032580] Lustre: Mounted lustre-client - version 2.17.58_3_g357faf0
[ 5152.693198] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 02:43:18 (1788590598)
[ 5155.188713] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5156.846129] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 02:43:22 (1788590602)
[ 5188.098449] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 02:43:54 (1788590634)
[ 5190.355363] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5192.542170] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 02:43:58 (1788590638)
[ 5194.231768] Lustre: *** cfs_fail_loc=805, val=0***
[ 5194.318790] Lustre: Mounted lustre-client - version 2.17.58_3_g357faf0
[ 5200.382635] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5219.807314] Lustre: 2359:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788590651/real 1788590651]  req@ffff9c7446d6dc00 x1875467874806400/t0(0) o400->lustre-MDT0000-mdc-ffff9c74513e1000@192.168.203.133@tcp:12/10 lens 224/224 e 0 to 1 dl 1788590667 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5219.808921] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 192.168.203.133@tcp) was lost; in progress operations using this service will fail
[ 5219.838202] Lustre: 2359:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 5230.056322] Lustre: Evicted from MGS (at 192.168.203.133@tcp) after server handle changed from 0x70985678e72f6213 to 0x70985678e72f652a
[ 5238.678682] Lustre: Unmounted lustre-client
[ 5247.121403] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 02:44:52 (1788590692)
[ 5289.450717] LustreError: 2355:0:(client.c:3449:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff9c744463b100 x1875467874818688/t90194313221(90194313221) o101->lustre-MDT0000-mdc-ffff9c74513e1000@192.168.203.133@tcp:12/10 lens 648/608 e 0 to 0 dl 1788590752 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5297.286858] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5298.952292] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5307.393622] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 02:45:53 (1788590753)
[ 5309.240376] LustreError: lustre-OST0000-osc-ffff9c74513e1000: operation ost_write to node 192.168.203.133@tcp failed: rc = -107
[ 5309.256613] LustreError: Skipped 3 previous similar messages
[ 5309.283157] LustreError: lustre-OST0000-osc-ffff9c74513e1000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5309.303880] Lustre: 2359:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.133@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5309.363192] LustreError: 72821:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff9c74513e1000: namespace resource [0x240000400:0x3b02:0x0].0x0 (ffff9c7446eb8800) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5317.689384] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 02:46:03 (1788590763)
[ 5319.244217] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5321.268905] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 02:46:07 (1788590767)
[ 5322.860612] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5324.670220] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 02:46:10 (1788590770)
[ 5326.139976] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5327.918818] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 02:46:13 (1788590773)
[ 5329.714724] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5331.874527] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 02:46:17 (1788590777)
[ 5333.401845] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5335.059556] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 02:46:21 (1788590781)
[ 5336.784156] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5338.843520] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 02:46:24 (1788590784)
[ 5340.563291] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5342.492962] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 02:46:28 (1788590788)
[ 5344.230782] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5345.717470] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 02:46:31 (1788590791)
[ 5347.286553] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5349.313371] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 02:46:35 (1788590795)
[ 5350.782473] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5352.629800] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 02:46:38 (1788590798)
[ 5354.132593] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5355.968966] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 02:46:41 (1788590801)
[ 5358.888449] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5360.906356] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 02:46:46 (1788590806)
[ 5392.282877] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 02:47:18 (1788590838)
[ 5420.939656] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 02:47:47 (1788590867)
[ 5421.261555] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5430.577657] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 02:47:56 (1788590876)
[ 5430.848069] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5440.782279] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 02:48:07 (1788590887)
[ 5441.277867] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5450.948727] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 02:48:16 (1788590896)
[ 5451.227673] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5462.261749] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 02:48:28 (1788590908)
[ 5462.605387] Lustre: *** cfs_fail_loc=510, val=3***
[ 5471.879723] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 02:48:37 (1788590917)
[ 5472.392242] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5531.103424] Lustre: lustre-OST0000-osc-ffff9c74513e1000: Connection to lustre-OST0000 (at 192.168.203.133@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5531.123569] Lustre: Skipped 16 previous similar messages
[ 5537.401981] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 02:49:43 (1788590983)
[ 5538.025336] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5604.106792] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 02:50:50 (1788591050)
[ 5604.478501] LustreError: 83038:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5607.272891] LustreError: lustre-MDT0000-mdc-ffff9c74513e1000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5607.293755] LustreError: 83053:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9c74513e1000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9c7445182f00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5607.339952] Lustre: lustre-MDT0000-mdc-ffff9c74513e1000: Connection restored to 192.168.203.133@tcp (at 192.168.203.133@tcp)
[ 5607.354936] Lustre: Skipped 16 previous similar messages
[ 5608.495134] LustreError: 83038:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5610.618540] LustreError: 83068:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5613.503781] LustreError: lustre-MDT0000-mdc-ffff9c74513e1000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5613.535595] LustreError: 83083:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9c74513e1000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9c7450c1da00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5613.556041] LustreError: 83083:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5614.679341] LustreError: 83068:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5621.437141] LustreError: lustre-MDT0000-mdc-ffff9c74513e1000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5621.449670] LustreError: 83112:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9c74513e1000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff9c7445182500) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5621.470880] LustreError: 83112:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5621.816928] LustreError: 83125:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5624.561516] LustreError: lustre-MDT0000-mdc-ffff9c74513e1000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5624.612490] LustreError: 83144:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5624.618434] LustreError: 83144:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5625.693346] LustreError: 83150:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5625.705857] LustreError: 83150:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5625.888192] LustreError: 83125:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5625.894941] LustreError: 83125:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9c74513e1000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5628.031988] LustreError: 83173:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5628.040649] LustreError: 83173:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5631.612889] LustreError: 83200:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5634.394611] LustreError: lustre-MDT0000-mdc-ffff9c74513e1000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5634.413739] LustreError: 83214:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9c74513e1000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9c7450c1db00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5635.679272] LustreError: 83200:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5642.436262] LustreError: lustre-MDT0000-mdc-ffff9c74513e1000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5642.829364] LustreError: 83256:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5645.826901] LustreError: lustre-MDT0000-mdc-ffff9c74513e1000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5645.850753] LustreError: 83270:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9c74513e1000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9c7445182400) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5645.888621] LustreError: 83270:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5646.831131] LustreError: 83256:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5649.724620] LustreError: lustre-MDT0000-mdc-ffff9c74513e1000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5649.780513] LustreError: 83303:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5649.785873] LustreError: 83303:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5651.033830] LustreError: 83284:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9c74513e1000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5658.804711] LustreError: 83355:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5658.820590] LustreError: 83355:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5662.377542] LustreError: lustre-MDT0000-mdc-ffff9c74513e1000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5662.479288] LustreError: 83378:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5662.487202] LustreError: 83378:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5662.887218] LustreError: 83355:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5662.896884] LustreError: 83355:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5662.917121] LustreError: 83355:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9c74513e1000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5674.408238] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 02:52:00 (1788591120)
[ 5694.052088] LustreError: 68277:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5694.060693] LustreError: 68277:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5698.151244] LustreError: 68277:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5698.165393] LustreError: 68277:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5706.189963] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 02:52:31 (1788591151)
[ 5756.513063] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 02:53:22 (1788591202)
[ 5819.309797] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 02:54:25 (1788591265)
[ 5851.412686] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 02:54:57 (1788591297)
[ 5851.832101] Lustre: Mounted lustre-client - version 2.17.58_3_g357faf0
[ 5974.730789] Lustre: Unmounted lustre-client
[ 5981.403130] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 02:57:07 (1788591427)
[ 5981.944764] LustreError: 87681:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5984.549929] LustreError: lustre-OST0000-osc-ffff9c74513e1000: operation ldlm_enqueue to node 192.168.203.133@tcp failed: rc = -107
[ 5984.562574] LustreError: Skipped 9 previous similar messages
[ 5984.571194] LustreError: lustre-OST0000-osc-ffff9c74513e1000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5984.586230] LustreError: 87763:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff9c74513e1000: namespace resource [0x240000400:0x3b2d:0x0].0x0 (ffff9c7450c1d300) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5984.611617] LustreError: 87763:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5984.999129] LustreError: 87681:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5985.005040] Lustre: 2358:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.133@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5991.742699] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 02:57:17 (1788591437)
[ 6048.736486] Lustre: 88349:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788591441/real 1788591441]  req@ffff9c7446d6e680 x1875467874995072/t0(0) o101->lustre-MDT0000-mdc-ffff9c74513e1000@192.168.203.133@tcp:12/10 lens 328/344 e 0 to 1 dl 1788591496 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 6048.763433] Lustre: 88349:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 18 previous similar messages
[ 6056.747571] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 02:58:22 (1788591502)
[ 6058.578259] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 6060.568123] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 02:58:26 (1788591506)
[ 6123.680948] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6125.049573] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 02:59:31 (1788591571)
[ 6180.831984] Lustre: lustre-MDT0000-mdc-ffff9c74513e1000: Connection to lustre-MDT0000 (at 192.168.203.133@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6180.851797] Lustre: Skipped 16 previous similar messages
[ 6182.957197] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 03:00:28 (1788591628)
[ 6185.633529] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6187.594299] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 03:00:33 (1788591633)
[ 6189.383405] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6191.521317] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 03:00:37 (1788591637)
[ 6228.589930] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 03:01:13 (1788591673)
[ 6239.390512] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6262.751741] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 192.168.203.133@tcp) was lost; in progress operations using this service will fail
[ 6262.764475] LustreError: Skipped 2 previous similar messages
[ 6268.019168] Lustre: lustre-MDT0000-mdc-ffff9c74513e1000: Connection restored to 192.168.203.133@tcp (at 192.168.203.133@tcp)
[ 6268.028796] Lustre: Skipped 14 previous similar messages
[ 6273.071273] Lustre: Evicted from MGS (at 192.168.203.133@tcp) after server handle changed from 0x70985678e72f7052 to 0x70985678e72f8b2c
[ 6273.081320] Lustre: Skipped 2 previous similar messages
[ 6276.959432] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6278.253321] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6286.454843] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 03:02:12 (1788591732)
[ 6288.295286] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6290.105151] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 03:02:16 (1788591736)
[ 6303.722653] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 192.168.203.133@tcp) was lost; in progress operations using this service will fail
[ 6314.470890] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 03:02:40 (1788591760)
[ 6344.752838] Lustre: Evicted from MGS (at 192.168.203.133@tcp) after server handle changed from 0x70985678e72f8d39 to 0x70985678e72f9295
[ 6344.776498] Lustre: Skipped 1 previous similar message
[ 6358.000126] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 03:03:23 (1788591803)
[ 6417.511491] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6423.563289] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6509.343441] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 192.168.203.133@tcp) was lost; in progress operations using this service will fail
[ 6509.351418] LustreError: Skipped 1 previous similar message
[ 6509.363513] Lustre: Evicted from MGS (at 192.168.203.133@tcp) after server handle changed from 0x70985678e72f9295 to 0x70985678e72f9d93
[ 6510.899505] Lustre: 68279:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.133@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6532.512178] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6534.079356] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6558.578801] Lustre: 68279:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.133@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6579.880588] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6581.289628] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6692.944215] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 03:08:59 (1788592139)
[ 6703.149806] LustreError: lustre-OST0000-osc-ffff9c74513e1000: operation ost_setattr to node 192.168.203.133@tcp failed: rc = -19
[ 6703.168359] LustreError: Skipped 33 previous similar messages
[ 6753.110388] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6756.862229] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6994.859881] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 03:14:01 (1788592441)
[ 7149.555884] Lustre: lustre-MDT0000-mdc-ffff9c74513e1000: Connection to lustre-MDT0000 (at 192.168.203.133@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7149.589766] Lustre: Skipped 7 previous similar messages
[ 7149.597986] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 192.168.203.133@tcp) was lost; in progress operations using this service will fail
[ 7149.615403] LustreError: Skipped 1 previous similar message
[ 7149.638278] Lustre: Evicted from MGS (at 192.168.203.133@tcp) after server handle changed from 0x70985678e72fa1a6 to 0x70985678e73f4433
[ 7149.652257] Lustre: Skipped 1 previous similar message
[ 7149.668849] Lustre: MGC192.168.203.133@tcp: Connection restored to 192.168.203.133@tcp (at 192.168.203.133@tcp)
[ 7149.681600] Lustre: Skipped 9 previous similar messages
[ 7180.986904] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 03:17:07 (1788592627)
[ 7182.310828] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7184.036351] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 03:17:10 (1788592630)
[ 7185.731399] LustreError: lustre-MDT0000-mdc-ffff9c74513e1000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7193.044949] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 03:17:18 (1788592638)
[ 7360.177681] LustreError: lustre-OST0000-osc-ffff9c74513e1000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7361.481811] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 03:20:07 (1788592807)
[ 7385.567159] Lustre: 2356:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788592812/real 1788592812]  req@ffff9c74485f8380 x1875467892733440/t0(0) o4->lustre-OST0000-osc-ffff9c74513e1000@192.168.203.133@tcp:6/4 lens 4584/448 e 0 to 1 dl 1788592832 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7385.585072] Lustre: 2356:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 30 previous similar messages
[ 7401.521405] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 03:20:47 (1788592847)
[ 7402.807057] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7404.311061] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 03:20:50 (1788592850)
[ 7405.687818] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7407.278118] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 03:20:53 (1788592853)
[ 7439.822511] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 03:21:25 (1788592885)
[ 7477.747967] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 192.168.203.133@tcp) was lost; in progress operations using this service will fail
[ 7477.823863] Lustre: Evicted from MGS (at 192.168.203.133@tcp) after server handle changed from 0x70985678e73f4433 to 0x70985678e73f6e6b
[ 7477.849217] Lustre: Skipped 1 previous similar message
[ 7497.562884] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 03:22:23 (1788592943)
[ 7499.519923] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7501.295466] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 03:22:27 (1788592947)
[ 7502.955079] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7505.090243] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 03:22:30 (1788592950)
[ 7513.047337] Lustre: Unmounted lustre-client
[ 7514.693468] Lustre: Mounted lustre-client - version 2.17.58_3_g357faf0
[ 7517.942492] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7549.253656] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2