[ 1819.634963] LustreError: lustre-OST0000-osc-ffff96cfd04f6000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1819.655916] Lustre: 2349:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.206.125@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -5)
[ 1821.923102] Lustre: DEBUG MARKER: recovery-small test_24b: @@@@@@ IGNORE (bz5494): multiop didn't fail fsync: 0 or close: 5
[ 1831.413874] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 20:10:33 (1788653433)
[ 1834.424771] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1836.082909] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 20:10:38 (1788653438)
[ 1838.194750] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1840.108894] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 20:10:41 (1788653441)
[ 1843.459457] LustreError: lustre-MDT0000-mdc-ffff96cfd04f6000: operation mds_reint to node 192.168.206.125@tcp failed: rc = -19
[ 1843.473896] LustreError: Skipped 5 previous similar messages
[ 1863.660039] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 192.168.206.125@tcp) was lost; in progress operations using this service will fail
[ 1863.702168] Lustre: Evicted from MGS (at 192.168.206.125@tcp) after server handle changed from 0x30846715838eb15c to 0x30846715838ecbe9
[ 1964.999159] LustreError: lustre-MDT0000-mdc-ffff96cfd04f6000: operation ldlm_enqueue to node 192.168.206.125@tcp failed: rc = -19
[ 1982.432829] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 192.168.206.125@tcp) was lost; in progress operations using this service will fail
[ 1982.455767] Lustre: Evicted from MGS (at 192.168.206.125@tcp) after server handle changed from 0x30846715838ecbe9 to 0x3084671583915a5b
[ 2007.168234] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 20:13:29 (1788653609)
[ 2007.604743] Lustre: *** cfs_fail_loc=305, val=0***
[ 2007.609829] Lustre: Skipped 2 previous similar messages
[ 2052.584297] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 192.168.206.125@tcp) was lost; in progress operations using this service will fail
[ 2052.618423] Lustre: Evicted from MGS (at 192.168.206.125@tcp) after server handle changed from 0x3084671583915a5b to 0x3084671583916203
[ 2063.227825] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2064.774772] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2072.595574] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 20:14:34 (1788653674)
[ 2083.306553] Lustre: lustre-MDT0000-mdc-ffff96cfd04f6000: Connection to lustre-MDT0000 (at 192.168.206.125@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2083.312832] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 192.168.206.125@tcp) was lost; in progress operations using this service will fail
[ 2083.325899] Lustre: Skipped 12 previous similar messages
[ 2083.354691] Lustre: Evicted from MGS (at 192.168.206.125@tcp) after server handle changed from 0x3084671583916203 to 0x3084671583916552
[ 2083.367563] Lustre: MGC192.168.206.125@tcp: Connection restored to 192.168.206.125@tcp (at 192.168.206.125@tcp)
[ 2083.377585] Lustre: Skipped 15 previous similar messages
[ 2085.061248] LustreError: lustre-MDT0000-mdc-ffff96cfd04f6000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2103.801348] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 20:15:05 (1788653705)
[ 2130.552461] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 20:15:32 (1788653732)
[ 2142.873433] LustreError: lustre-MDT0000-mdc-ffff96cfd04f6000: operation ldlm_enqueue to node 192.168.206.125@tcp failed: rc = -19
[ 2142.878901] LustreError: Skipped 1 previous similar message
[ 2161.120228] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 192.168.206.125@tcp) was lost; in progress operations using this service will fail
[ 2171.323579] Lustre: Evicted from MGS (at 192.168.206.125@tcp) after server handle changed from 0x3084671583916552 to 0x308467158391c4e3
[ 2186.299794] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2187.904164] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2272.746650] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 192.168.206.125@tcp) was lost; in progress operations using this service will fail
[ 2272.781205] Lustre: Evicted from MGS (at 192.168.206.125@tcp) after server handle changed from 0x308467158391c4e3 to 0x3084671583939a53
[ 2284.324540] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2285.864160] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2350.683311] LustreError: lustre-MDT0000-mdc-ffff96cfd04f6000: operation ldlm_enqueue to node 192.168.206.125@tcp failed: rc = -19
[ 2350.696497] LustreError: Skipped 1 previous similar message
[ 2369.952146] Lustre: 2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788653957/real 1788653957]  req@ffff96cfc5be0380 x1875537220923776/t0(0) o400->MGC192.168.206.125@tcp@192.168.206.125@tcp:26/25 lens 224/224 e 0 to 1 dl 1788653973 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2369.980952] Lustre: 2352:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 10 previous similar messages
[ 2369.989571] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 192.168.206.125@tcp) was lost; in progress operations using this service will fail
[ 2370.012617] Lustre: Evicted from MGS (at 192.168.206.125@tcp) after server handle changed from 0x3084671583939a53 to 0x308467158395b951
[ 2371.439927] Lustre: 15910:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.125@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2395.437561] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2397.329201] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2425.547246] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 20:20:27 (1788654027)
[ 2445.792357] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 192.168.206.125@tcp) was lost; in progress operations using this service will fail
[ 2454.128836] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2455.993855] Lustre: Evicted from MGS (at 192.168.206.125@tcp) after server handle changed from 0x308467158395b951 to 0x308467158396972f
[ 2472.928840] Lustre: 15910:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.125@tcp: IR log lustre-cliir failed, not fatal: rc = -5
[ 2481.518900] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2512.764420] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2547.432453] Lustre: 15910:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.125@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2561.217518] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2600.928232] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 192.168.206.125@tcp) was lost; in progress operations using this service will fail
[ 2600.941749] LustreError: Skipped 3 previous similar messages
[ 2609.207859] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2611.186044] Lustre: Evicted from MGS (at 192.168.206.125@tcp) after server handle changed from 0x308467158396e57f to 0x308467158397747a
[ 2611.198224] Lustre: Skipped 3 previous similar messages
[ 2637.970547] LustreError: lustre-MDT0000-mdc-ffff96cfd04f6000: operation mds_reint to node 192.168.206.125@tcp failed: rc = -19
[ 2637.986663] LustreError: Skipped 4 previous similar messages
[ 2676.820356] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2709.877528] Lustre: lustre-MDT0000-mdc-ffff96cfd04f6000: Connection to lustre-MDT0000 (at 192.168.206.125@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2709.895448] Lustre: Skipped 8 previous similar messages
[ 2727.934078] Lustre: MGC192.168.206.125@tcp: Connection restored to 192.168.206.125@tcp (at 192.168.206.125@tcp)
[ 2727.946797] Lustre: Skipped 17 previous similar messages
[ 2729.650320] Lustre: 15910:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.125@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2769.092262] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 20:26:10 (1788654370)
[ 2815.403873] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 2817.321694] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3145.150766] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3146.687112] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3445.886442] LustreError: lustre-OST0000-osc-ffff96cfd04f6000: operation ldlm_enqueue to node 192.168.206.125@tcp failed: rc = -107
[ 3445.898417] LustreError: Skipped 3 previous similar messages
[ 3445.903771] Lustre: lustre-OST0000-osc-ffff96cfd04f6000: Connection to lustre-OST0000 (at 192.168.206.125@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3445.916511] Lustre: Skipped 2 previous similar messages
[ 3468.008632] Lustre: lustre-OST0000-osc-ffff96cfd04f6000: Connection restored to 192.168.206.125@tcp (at 192.168.206.125@tcp)
[ 3468.017599] Lustre: Skipped 2 previous similar messages
[ 3482.695980] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3484.673431] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3739.240850] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 20:42:21 (1788655341)
[ 3756.000195] Lustre: 50851:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788655343/real 1788655343]  req@ffff96cfc5b53800 x1875537245089664/t0(0) o101->lustre-MDT0000-mdc-ffff96cfd04f6000@192.168.206.125@tcp:12/10 lens 576/1152 e 0 to 1 dl 1788655359 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3756.018161] Lustre: 50851:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
[ 3763.428081] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 20:42:44 (1788655364)
[ 3787.648945] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 20:43:09 (1788655389)
[ 3812.924315] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 20:43:34 (1788655414)
[ 3813.470405] Lustre: Mounted lustre-client - version 2.17.58_2_g557ce73
[ 3844.065591] Lustre: 2351:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788655431/real 1788655431]  req@ffff96cfc36cc380 x1875537245108480/t0(0) o400->lustre-MDT0000-mdc-ffff96cfd04f6000@192.168.206.125@tcp:12/10 lens 224/224 e 0 to 1 dl 1788655447 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3844.082443] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 192.168.206.125@tcp) was lost; in progress operations using this service will fail
[ 3844.094613] Lustre: 2351:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 3844.109373] LustreError: Skipped 2 previous similar messages
[ 3844.119514] Lustre: Evicted from MGS (at 192.168.206.125@tcp) after server handle changed from 0x308467158398cde9 to 0x3084671583b2251b
[ 3844.127354] Lustre: Skipped 2 previous similar messages
[ 3857.719569] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 3859.091301] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3861.106423] Lustre: Unmounted lustre-client
[ 3866.407338] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 20:44:28 (1788655468)
[ 4002.784356] Lustre: 2351:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788655590/real 1788655590]  req@ffff96cfc36cfb80 x1875537245141376/t0(0) o4->lustre-OST0000-osc-ffff96cfd04f6000@192.168.206.125@tcp:6/4 lens 488/448 e 0 to 1 dl 1788655606 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4002.817462] Lustre: 2351:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 47 previous similar messages
[ 4050.912325] Lustre: lustre-OST0000-osc-ffff96cfd04f6000: Connection to lustre-OST0000 (at 192.168.206.125@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4050.930245] Lustre: Skipped 15 previous similar messages
[ 4083.696577] Lustre: lustre-OST0000-osc-ffff96cfd04f6000: Connection restored to 192.168.206.125@tcp (at 192.168.206.125@tcp)
[ 4083.713212] Lustre: Skipped 18 previous similar messages
[ 4159.566180] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 20:49:21 (1788655761)
[ 4207.554759] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 20:50:09 (1788655809)
[ 4211.290319] Lustre: Unmounted lustre-client
[ 4233.571600] Lustre: Mounted lustre-client - version 2.17.58_2_g557ce73
[ 4240.059805] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 20:50:41 (1788655841)
[ 4240.318323] LustreError: 56772:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4241.372021] LustreError: 56772:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4241.566768] Lustre: *** cfs_fail_loc=305, val=0***
[ 4264.595437] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 20:51:05 (1788655865)
[ 4265.397864] Lustre: Mounted lustre-client - version 2.17.58_2_g557ce73
[ 4268.379503] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4278.821613] Lustre: Unmounted lustre-client
[ 4290.402692] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 20:51:31 (1788655891)
[ 4373.063410] LustreError: lustre-MDT0000-mdc-ffff96cfd163e000: operation mds_reint to node 192.168.206.125@tcp failed: rc = -19
[ 4390.368212] Lustre: 2350:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788655977/real 1788655977]  req@ffff96cfc8880e00 x1875537247822592/t0(0) o400->MGC192.168.206.125@tcp@192.168.206.125@tcp:26/25 lens 224/224 e 0 to 1 dl 1788655993 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4390.423023] Lustre: 2350:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 85 previous similar messages
[ 4390.444351] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 192.168.206.125@tcp) was lost; in progress operations using this service will fail
[ 4400.056204] Lustre: Evicted from MGS (at 192.168.206.125@tcp) after server handle changed from 0x3084671583b2416f to 0x3084671583b52856
[ 4521.180480] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 20:55:23 (1788656123)
[ 4525.513282] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4538.345901] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 192.168.206.125@tcp) was lost; in progress operations using this service will fail
[ 4538.372867] Lustre: Evicted from MGS (at 192.168.206.125@tcp) after server handle changed from 0x3084671583b52856 to 0x3084671583b9bf30
[ 4548.618265] LustreError: lustre-MDT0000-mdc-ffff96cfd163e000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4558.042338] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 20:55:59 (1788656159)
[ 4558.474798] Lustre: Mounted lustre-client - version 2.17.58_2_g557ce73
[ 4565.855622] LustreError: lustre-OST0000-osc-ffff96cfd163e000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4578.678782] Lustre: Unmounted lustre-client
[ 4585.512047] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 20:56:27 (1788656187)
[ 4592.408166] LustreError: lustre-MDT0000-mdc-ffff96cfd163e000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4592.429307] LustreError: 61159:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4599.543705] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 20:56:41 (1788656201)
[ 4599.723378] LustreError: 61857:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4603.841672] LustreError: lustre-MDT0000-mdc-ffff96cfd163e000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4603.860082] LustreError: 61880:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4603.882944] LustreError: 61857:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=850
[ 4603.901522] LustreError: 61857:0:(import.c:761:ptlrpc_connect_import_locked()) already connecting
[ 4603.966066] LustreError: 61885:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4605.062827] LustreError: 61892:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4605.081545] LustreError: 61892:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4607.382974] LustreError: 61914:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4607.395398] LustreError: 61914:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4612.077944] LustreError: 61959:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4612.093211] LustreError: 61959:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4621.349208] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 20:57:03 (1788656223)
[ 4660.134320] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4661.301238] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4670.615824] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 20:57:52 (1788656272)
[ 4675.566379] Lustre: lustre-OST0000-osc-ffff96cfd163e000: Connection to lustre-OST0000 (at 192.168.206.125@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4675.582066] Lustre: Skipped 18 previous similar messages
[ 4695.417553] Lustre: lustre-OST0000-osc-ffff96cfd163e000: Connection restored to 192.168.206.125@tcp (at 192.168.206.125@tcp)
[ 4695.428453] Lustre: Skipped 16 previous similar messages
[ 4711.025608] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4712.757598] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4722.923372] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 20:58:44 (1788656324)
[ 4787.029917] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4788.537666] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4796.330819] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 20:59:58 (1788656398)
[ 4837.545663] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4839.422347] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4847.465080] Lustre: Unmounted lustre-client
[ 4903.403194] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4905.894193] Lustre: Mounted lustre-client - version 2.17.58_2_g557ce73
[ 4911.552167] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 21:01:53 (1788656513)
[ 4913.893850] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 4915.472863] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 21:01:57 (1788656517)
[ 4945.406016] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 21:02:27 (1788656547)
[ 4946.843224] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 4948.543467] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 21:02:30 (1788656550)
[ 4949.945514] Lustre: *** cfs_fail_loc=805, val=0***
[ 4949.983778] Lustre: Mounted lustre-client - version 2.17.58_2_g557ce73
[ 4955.842505] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4976.613682] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 192.168.206.125@tcp) was lost; in progress operations using this service will fail
[ 4985.838963] Lustre: Evicted from MGS (at 192.168.206.125@tcp) after server handle changed from 0x3084671583b9c9c5 to 0x3084671583b9cd6f
[ 4985.861755] LustreError: 2348:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff96cfd02c4800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff96cfc9566a00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4988.293924] Lustre: Unmounted lustre-client
[ 4992.096105] Lustre: 2349:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788656578/real 1788656578]  req@ffff96cfc8324380 x1875537249870336/t0(0) o400->lustre-MDT0000-mdc-ffff96cfc368c000@192.168.206.125@tcp:12/10 lens 224/224 e 0 to 1 dl 1788656594 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4992.131300] Lustre: 2349:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 9 previous similar messages
[ 4996.307562] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 21:03:18 (1788656598)
[ 5023.256451] LustreError: 2348:0:(client.c:3449:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff96cfc8325180 x1875537249879040/t85899345924(85899345924) o101->lustre-MDT0000-mdc-ffff96cfc368c000@192.168.206.125@tcp:12/10 lens 664/608 e 0 to 0 dl 1788656642 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5033.069350] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5034.583261] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5043.145198] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 21:04:05 (1788656645)
[ 5044.186751] LustreError: lustre-OST0000-osc-ffff96cfc368c000: operation ost_write to node 192.168.206.125@tcp failed: rc = -107
[ 5044.203554] LustreError: Skipped 3 previous similar messages
[ 5044.227576] LustreError: lustre-OST0000-osc-ffff96cfc368c000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5044.249163] Lustre: 2352:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.206.125@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5044.293864] LustreError: 73076:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff96cfc368c000: namespace resource [0x240000400:0x4882:0x0].0x0 (ffff96cfc706ca00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5044.313611] LustreError: 73076:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5052.805498] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 21:04:14 (1788656654)
[ 5054.339309] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5056.109893] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 21:04:18 (1788656658)
[ 5057.646705] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5060.291304] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 21:04:21 (1788656661)
[ 5062.102860] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5063.926583] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 21:04:25 (1788656665)
[ 5065.141497] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5066.648105] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 21:04:28 (1788656668)
[ 5068.015968] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5070.258978] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 21:04:31 (1788656671)
[ 5071.407669] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5072.861558] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 21:04:34 (1788656674)
[ 5074.058578] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5075.521236] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 21:04:37 (1788656677)
[ 5076.777687] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5078.471342] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 21:04:40 (1788656680)
[ 5079.990539] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5081.339914] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 21:04:43 (1788656683)
[ 5083.154316] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5084.741564] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 21:04:46 (1788656686)
[ 5086.114345] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5087.888583] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 21:04:49 (1788656689)
[ 5090.208971] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5091.825470] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 21:04:53 (1788656693)
[ 5124.422759] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 21:05:26 (1788656726)
[ 5153.717989] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 21:05:55 (1788656755)
[ 5154.067480] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5164.324194] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 21:06:06 (1788656766)
[ 5164.652706] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5175.645472] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 21:06:17 (1788656777)
[ 5176.007690] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5184.451392] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 21:06:26 (1788656786)
[ 5185.023631] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5193.203716] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 21:06:35 (1788656795)
[ 5193.494528] Lustre: *** cfs_fail_loc=510, val=3***
[ 5202.310921] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 21:06:44 (1788656804)
[ 5202.660155] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5212.171424] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 21:06:54 (1788656814)
[ 5212.535889] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5278.950079] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 21:08:00 (1788656880)
[ 5279.441103] LustreError: 83280:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5282.561448] Lustre: lustre-MDT0000-mdc-ffff96cfc368c000: Connection to lustre-MDT0000 (at 192.168.206.125@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5282.575098] Lustre: Skipped 9 previous similar messages
[ 5282.598735] LustreError: lustre-MDT0000-mdc-ffff96cfc368c000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5282.641676] LustreError: 83294:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff96cfc368c000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff96cfc9566900) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5283.520170] LustreError: 83280:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5285.661424] LustreError: 83310:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5288.573238] LustreError: lustre-MDT0000-mdc-ffff96cfc368c000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5288.596697] LustreError: 83324:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff96cfc368c000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff96cfc7070700) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5288.633877] LustreError: 83324:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5289.737275] LustreError: 83310:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5296.510942] LustreError: lustre-MDT0000-mdc-ffff96cfc368c000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5296.537155] Lustre: lustre-MDT0000-mdc-ffff96cfc368c000: Connection restored to 192.168.206.125@tcp (at 192.168.206.125@tcp)
[ 5296.552436] Lustre: Skipped 12 previous similar messages
[ 5296.893298] LustreError: 83366:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5299.853451] LustreError: lustre-MDT0000-mdc-ffff96cfc368c000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5299.946794] LustreError: 83386:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5299.953256] LustreError: 83386:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5300.968110] LustreError: 83366:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5300.984197] LustreError: 83366:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff96cfc368c000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5301.065229] LustreError: 83392:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5301.079297] LustreError: 83392:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5303.386351] LustreError: 83414:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5303.408391] LustreError: 83414:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5307.100356] LustreError: 83441:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5310.105591] LustreError: lustre-MDT0000-mdc-ffff96cfc368c000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5310.122241] LustreError: 83456:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff96cfc368c000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff96cfc7ae0700) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5310.152649] LustreError: 83456:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5311.184127] LustreError: 83441:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5317.901888] LustreError: lustre-MDT0000-mdc-ffff96cfc368c000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5318.250187] LustreError: 83497:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5321.155294] LustreError: lustre-MDT0000-mdc-ffff96cfc368c000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5321.172852] LustreError: 83512:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff96cfc368c000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff96cfc7070300) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5321.197927] LustreError: 83512:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5322.320127] LustreError: 83497:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5325.338430] LustreError: lustre-MDT0000-mdc-ffff96cfc368c000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5325.409072] LustreError: 83545:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5325.422092] LustreError: 83545:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5326.529104] LustreError: 83525:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff96cfc368c000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5334.384253] LustreError: 83597:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5334.393861] LustreError: 83597:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5337.791967] LustreError: lustre-MDT0000-mdc-ffff96cfc368c000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5337.872682] LustreError: 83619:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5337.880607] LustreError: 83619:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5338.472632] LustreError: 83597:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5338.479429] LustreError: 83597:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5338.494066] LustreError: 83597:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff96cfc368c000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5348.982405] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 21:09:10 (1788656950)
[ 5367.382341] LustreError: 68533:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5367.392283] LustreError: 68533:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5371.464120] LustreError: 68533:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5371.471082] LustreError: 68533:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5377.711743] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 21:09:39 (1788656979)
[ 5429.920700] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 21:10:31 (1788657031)
[ 5493.990157] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 21:11:35 (1788657095)
[ 5526.964666] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 21:12:09 (1788657129)
[ 5527.468326] Lustre: Mounted lustre-client - version 2.17.58_2_g557ce73
[ 5650.717962] Lustre: Unmounted lustre-client
[ 5657.789321] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 21:14:19 (1788657259)
[ 5658.332306] LustreError: 87924:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5660.653710] LustreError: lustre-OST0000-osc-ffff96cfc368c000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5660.680354] LustreError: 88006:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff96cfc368c000: namespace resource [0x240000400:0x48ad:0x0].0x0 (ffff96cfc706c600) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5660.704308] LustreError: 88006:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5660.984124] LustreError: 87924:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5660.989980] Lustre: 2352:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.206.125@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5667.092960] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 21:14:29 (1788657269)
[ 5724.128159] Lustre: 88590:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788657272/real 1788657272]  req@ffff96cfc83bad80 x1875537250052096/t0(0) o101->lustre-MDT0000-mdc-ffff96cfc368c000@192.168.206.125@tcp:12/10 lens 328/344 e 0 to 1 dl 1788657327 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 5724.173650] Lustre: 88590:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 12 previous similar messages
[ 5730.771539] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 21:15:32 (1788657332)
[ 5732.018595] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 5733.503580] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 21:15:35 (1788657335)
[ 5796.429348] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 5797.772426] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 21:16:40 (1788657400)
[ 5855.744394] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 21:17:37 (1788657457)
[ 5858.358907] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 5859.851774] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 21:17:41 (1788657461)
[ 5861.251841] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 5862.816908] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 21:17:44 (1788657464)
[ 5894.128677] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 21:18:16 (1788657496)
[ 5904.774102] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5930.976454] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 192.168.206.125@tcp) was lost; in progress operations using this service will fail
[ 5930.998819] LustreError: Skipped 2 previous similar messages
[ 5935.086659] Lustre: lustre-MDT0000-mdc-ffff96cfc368c000: Connection to lustre-MDT0000 (at 192.168.206.125@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5935.097503] Lustre: Skipped 15 previous similar messages
[ 5935.193218] Lustre: lustre-MDT0000-mdc-ffff96cfc368c000: Connection restored to 192.168.206.125@tcp (at 192.168.206.125@tcp)
[ 5935.208391] Lustre: Skipped 12 previous similar messages
[ 5941.286315] Lustre: Evicted from MGS (at 192.168.206.125@tcp) after server handle changed from 0x3084671583b9d80b to 0x3084671583b9f316
[ 5941.296654] Lustre: Skipped 2 previous similar messages
[ 5946.855437] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5948.527659] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5958.298321] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 21:19:20 (1788657560)
[ 5960.532806] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 5962.388854] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 21:19:24 (1788657564)
[ 5977.063795] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 192.168.206.125@tcp) was lost; in progress operations using this service will fail
[ 5977.087712] Lustre: Evicted from MGS (at 192.168.206.125@tcp) after server handle changed from 0x3084671583b9f316 to 0x3084671583b9f523
[ 5986.436099] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 21:19:48 (1788657588)
[ 6034.518605] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 21:20:36 (1788657636)
[ 6041.637496] LustreError: lustre-OST0000-osc-ffff96cfc368c000: operation ost_setattr to node 192.168.206.125@tcp failed: rc = -107
[ 6041.645348] LustreError: Skipped 9 previous similar messages
[ 6086.755551] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6090.032223] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6176.736364] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 192.168.206.125@tcp) was lost; in progress operations using this service will fail
[ 6176.751185] LustreError: Skipped 1 previous similar message
[ 6187.115441] Lustre: Evicted from MGS (at 192.168.206.125@tcp) after server handle changed from 0x3084671583b9f8aa to 0x3084671583ba0855
[ 6187.124411] Lustre: Skipped 1 previous similar message
[ 6194.592608] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6196.119610] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6230.128652] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6231.535484] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6361.362485] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 21:26:02 (1788657962)
[ 6421.119093] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6423.766578] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6669.906723] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 21:31:12 (1788658272)
[ 6781.428395] Lustre: lustre-MDT0000-mdc-ffff96cfc368c000: Connection to lustre-MDT0000 (at 192.168.206.125@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6781.442052] Lustre: Skipped 6 previous similar messages
[ 6791.652680] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 192.168.206.125@tcp) was lost; in progress operations using this service will fail
[ 6791.664958] LustreError: Skipped 1 previous similar message
[ 6791.690268] Lustre: Evicted from MGS (at 192.168.206.125@tcp) after server handle changed from 0x3084671583ba09a5 to 0x3084671583c9ac9b
[ 6791.698723] Lustre: Skipped 1 previous similar message
[ 6791.713208] Lustre: MGC192.168.206.125@tcp: Connection restored to 192.168.206.125@tcp (at 192.168.206.125@tcp)
[ 6791.721773] Lustre: Skipped 9 previous similar messages
[ 6829.142520] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 21:33:51 (1788658431)
[ 6830.252710] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 6831.667355] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 21:33:53 (1788658433)
[ 6833.319805] LustreError: lustre-MDT0000-mdc-ffff96cfc368c000: operation ldlm_enqueue to node 192.168.206.125@tcp failed: rc = -107
[ 6833.327128] LustreError: Skipped 405 previous similar messages
[ 6833.348821] LustreError: lustre-MDT0000-mdc-ffff96cfc368c000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6841.029766] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 21:34:02 (1788658442)
[ 7007.794473] LustreError: lustre-OST0000-osc-ffff96cfc368c000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7009.726862] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 21:36:51 (1788658611)
[ 7035.360204] Lustre: 2350:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788658618/real 1788658618]  req@ffff96cfeb56d500 x1875537267799936/t0(0) o4->lustre-OST0000-osc-ffff96cfc368c000@192.168.206.125@tcp:6/4 lens 4584/448 e 0 to 1 dl 1788658638 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7035.393957] Lustre: 2350:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 25 previous similar messages
[ 7054.111548] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 21:37:35 (1788658655)
[ 7055.642121] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7057.478595] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 21:37:39 (1788658659)
[ 7058.841734] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7060.901977] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 21:37:42 (1788658662)
[ 7095.207410] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 21:38:17 (1788658697)
[ 7135.724714] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 192.168.206.125@tcp) was lost; in progress operations using this service will fail
[ 7135.742817] Lustre: Evicted from MGS (at 192.168.206.125@tcp) after server handle changed from 0x3084671583c9ac9b to 0x3084671583c9d425
[ 7135.753904] Lustre: Skipped 1 previous similar message
[ 7151.770208] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 21:39:13 (1788658753)
[ 7153.730024] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7155.769238] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 21:39:17 (1788658757)
[ 7157.465070] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7159.089460] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 21:39:20 (1788658760)
[ 7168.239659] Lustre: Unmounted lustre-client
[ 7169.412287] Lustre: Mounted lustre-client - version 2.17.58_2_g557ce73
[ 7172.287058] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7189.984159] Lustre: 2349:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788658777/real 1788658777]  req@ffff96cfd0f48700 x1875537267987328/t0(0) o400->MGC192.168.206.125@tcp@192.168.206.125@tcp:26/25 lens 224/224 e 0 to 1 dl 1788658793 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7200.867984] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2