[ 2019.125086] LustreError: lustre-OST0000-osc-ffff988212389800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2019.145888] Lustre: 2347:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.122@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 2025.993406] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 19:48:18 (1786751298)
[ 2027.995524] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 2030.076969] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 19:48:22 (1786751302)
[ 2032.269495] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 2034.582175] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 19:48:26 (1786751306)
[ 2057.695524] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 192.168.204.122@tcp) was lost; in progress operations using this service will fail
[ 2066.881352] Lustre: Evicted from MGS (at 192.168.204.122@tcp) after server handle changed from 0x7693a547d10c1f9a to 0x7693a547d10c38a6
[ 2163.800605] LustreError: lustre-MDT0000-mdc-ffff988212389800: operation ldlm_enqueue to node 192.168.204.122@tcp failed: rc = -19
[ 2163.812043] LustreError: Skipped 3 previous similar messages
[ 2184.672151] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 192.168.204.122@tcp) was lost; in progress operations using this service will fail
[ 2184.707706] Lustre: Evicted from MGS (at 192.168.204.122@tcp) after server handle changed from 0x7693a547d10c38a6 to 0x7693a547d10dbc0e
[ 2186.234802] Lustre: 15941:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.204.122@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2211.312695] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 19:51:24 (1786751484)
[ 2212.019255] Lustre: *** cfs_fail_loc=305, val=0***
[ 2212.023321] Lustre: Skipped 4 previous similar messages
[ 2255.331311] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 192.168.204.122@tcp) was lost; in progress operations using this service will fail
[ 2265.591298] Lustre: Evicted from MGS (at 192.168.204.122@tcp) after server handle changed from 0x7693a547d10dbc0e to 0x7693a547d10dc3d2
[ 2278.375792] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2280.352730] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2290.189962] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 19:52:43 (1786751563)
[ 2306.546795] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 192.168.204.122@tcp) was lost; in progress operations using this service will fail
[ 2306.591270] Lustre: Evicted from MGS (at 192.168.204.122@tcp) after server handle changed from 0x7693a547d10dc3d2 to 0x7693a547d10dc855
[ 2306.645403] LustreError: lustre-MDT0000-mdc-ffff988212389800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2334.372555] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 19:53:26 (1786751606)
[ 2352.644923] LustreError: lustre-OST0000-osc-ffff988212389800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2373.064404] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 19:54:05 (1786751645)
[ 2387.494187] LustreError: lustre-MDT0000-mdc-ffff988212389800: operation mds_reint to node 192.168.204.122@tcp failed: rc = -19
[ 2387.514202] LustreError: Skipped 1 previous similar message
[ 2404.322361] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 192.168.204.122@tcp) was lost; in progress operations using this service will fail
[ 2414.508882] Lustre: Evicted from MGS (at 192.168.204.122@tcp) after server handle changed from 0x7693a547d10dc855 to 0x7693a547d10e0977
[ 2416.239031] Lustre: 15941:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.204.122@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2437.259814] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2439.526736] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2527.199283] Lustre: 2350:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786751785/real 1786751785]  req@ffff9882101d1c00 x1873542509696256/t0(0) o400->MGC192.168.204.122@tcp@192.168.204.122@tcp:26/25 lens 224/224 e 0 to 1 dl 1786751801 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2527.233915] Lustre: 2350:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 15 previous similar messages
[ 2527.240715] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 192.168.204.122@tcp) was lost; in progress operations using this service will fail
[ 2537.447203] Lustre: Evicted from MGS (at 192.168.204.122@tcp) after server handle changed from 0x7693a547d10e0977 to 0x7693a547d10f9e58
[ 2537.470958] Lustre: MGC192.168.204.122@tcp: Connection restored to 192.168.204.122@tcp (at 192.168.204.122@tcp)
[ 2537.482822] Lustre: Skipped 14 previous similar messages
[ 2555.539329] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2558.215719] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2624.333175] Lustre: lustre-MDT0000-mdc-ffff988212389800: Connection to lustre-MDT0000 (at 192.168.204.122@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2624.345549] Lustre: Skipped 10 previous similar messages
[ 2643.935431] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 192.168.204.122@tcp) was lost; in progress operations using this service will fail
[ 2654.149519] Lustre: Evicted from MGS (at 192.168.204.122@tcp) after server handle changed from 0x7693a547d10f9e58 to 0x7693a547d1111a3b
[ 2673.729775] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2676.332853] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2708.409258] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 19:59:41 (1786751981)
[ 2714.019659] LustreError: lustre-MDT0000-mdc-ffff988212389800: operation ldlm_enqueue to node 192.168.204.122@tcp failed: rc = -19
[ 2714.033052] LustreError: Skipped 4 previous similar messages
[ 2735.074507] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 192.168.204.122@tcp) was lost; in progress operations using this service will fail
[ 2745.412430] Lustre: Evicted from MGS (at 192.168.204.122@tcp) after server handle changed from 0x7693a547d1111a3b to 0x7693a547d111f334
[ 2747.069482] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2783.102443] Lustre: 15941:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.204.122@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2799.816322] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2846.120209] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2878.431818] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 192.168.204.122@tcp) was lost; in progress operations using this service will fail
[ 2878.451147] LustreError: Skipped 2 previous similar messages
[ 2888.627265] Lustre: Evicted from MGS (at 192.168.204.122@tcp) after server handle changed from 0x7693a547d112600f to 0x7693a547d11297c9
[ 2888.638249] Lustre: Skipped 2 previous similar messages
[ 2890.312095] Lustre: 15941:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.204.122@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2902.784223] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2949.357750] Lustre: 15941:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.204.122@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2970.574385] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 3040.427944] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 3147.066539] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 20:06:59 (1786752419)
[ 3188.561397] Lustre: lustre-OST0000-osc-ffff988212389800: Connection restored to 192.168.204.122@tcp (at 192.168.204.122@tcp)
[ 3188.569299] Lustre: Skipped 17 previous similar messages
[ 3210.558860] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3213.389756] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3494.250610] LustreError: lustre-OST0000-osc-ffff988212389800: operation ldlm_enqueue to node 192.168.204.122@tcp failed: rc = -19
[ 3494.278500] LustreError: Skipped 8 previous similar messages
[ 3494.294303] Lustre: lustre-OST0000-osc-ffff988212389800: Connection to lustre-OST0000 (at 192.168.204.122@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3494.318592] Lustre: Skipped 8 previous similar messages
[ 3543.601384] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3546.332889] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3854.385593] Lustre: lustre-OST0000-osc-ffff988212389800: Connection restored to 192.168.204.122@tcp (at 192.168.204.122@tcp)
[ 3854.395053] Lustre: Skipped 1 previous similar message
[ 3876.247768] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3879.048082] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4121.858960] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 20:23:14 (1786753394)
[ 4138.975879] Lustre: 50585:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786753397/real 1786753397]  req@ffff9882101e9880 x1873542527780224/t0(0) o101->lustre-MDT0000-mdc-ffff988212389800@192.168.204.122@tcp:12/10 lens 576/1152 e 0 to 1 dl 1786753413 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 4139.036991] Lustre: 50585:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
[ 4139.056736] Lustre: lustre-MDT0000-mdc-ffff988212389800: Connection to lustre-MDT0000 (at 192.168.204.122@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4139.092208] Lustre: Skipped 1 previous similar message
[ 4147.659065] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 20:23:40 (1786753420)
[ 4173.968039] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 20:24:06 (1786753446)
[ 4201.109232] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 20:24:33 (1786753473)
[ 4201.677847] Lustre: Mounted lustre-client - version 2.17.57_2_g67a826c
[ 4233.695139] Lustre: 2348:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786753491/real 1786753491]  req@ffff9882101ea680 x1873542527799040/t0(0) o400->lustre-MDT0000-mdc-ffff988212389800@192.168.204.122@tcp:12/10 lens 224/224 e 0 to 1 dl 1786753507 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4233.745345] Lustre: 2348:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 4233.769707] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 192.168.204.122@tcp) was lost; in progress operations using this service will fail
[ 4233.784038] LustreError: Skipped 3 previous similar messages
[ 4243.954819] Lustre: Evicted from MGS (at 192.168.204.122@tcp) after server handle changed from 0x7693a547d1145b7a to 0x7693a547d124bbd9
[ 4243.964026] Lustre: Skipped 3 previous similar messages
[ 4258.865346] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4261.123770] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4263.603605] Lustre: Unmounted lustre-client
[ 4271.433016] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 20:25:43 (1786753543)
[ 4397.535173] Lustre: 2350:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786753655/real 1786753655]  req@ffff988210069180 x1873542527833472/t0(0) o4->lustre-OST0000-osc-ffff988212389800@192.168.204.122@tcp:6/4 lens 488/448 e 0 to 1 dl 1786753671 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4397.585627] Lustre: 2350:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 61 previous similar messages
[ 4461.001083] Lustre: lustre-OST0000-osc-ffff988212389800: Connection restored to 192.168.204.122@tcp (at 192.168.204.122@tcp)
[ 4461.013146] Lustre: Skipped 16 previous similar messages
[ 4572.433122] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 20:30:45 (1786753845)
[ 4624.288078] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 20:31:36 (1786753896)
[ 4628.705293] Lustre: Unmounted lustre-client
[ 4659.160280] Lustre: Mounted lustre-client - version 2.17.57_2_g67a826c
[ 4667.026757] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 20:32:19 (1786753939)
[ 4667.430679] LustreError: 56514:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4668.479595] LustreError: 56514:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4668.724104] Lustre: *** cfs_fail_loc=305, val=0***
[ 4692.395632] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 20:32:44 (1786753964)
[ 4693.047191] Lustre: Mounted lustre-client - version 2.17.57_2_g67a826c
[ 4695.549707] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4705.890066] Lustre: Unmounted lustre-client
[ 4713.565825] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 20:33:06 (1786753986)
[ 4821.499227] LustreError: lustre-MDT0000-mdc-ffff988207e4d800: operation mds_reint to node 192.168.204.122@tcp failed: rc = -19
[ 4821.515696] LustreError: Skipped 1 previous similar message
[ 4821.521280] Lustre: lustre-MDT0000-mdc-ffff988207e4d800: Connection to lustre-MDT0000 (at 192.168.204.122@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4821.538625] Lustre: Skipped 23 previous similar messages
[ 4842.464893] Lustre: 2350:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786754100/real 1786754100]  req@ffff98820653ca80 x1873542530511744/t0(0) o400->MGC192.168.204.122@tcp@192.168.204.122@tcp:26/25 lens 224/224 e 0 to 1 dl 1786754116 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4842.499502] Lustre: 2350:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 75 previous similar messages
[ 4842.523610] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 192.168.204.122@tcp) was lost; in progress operations using this service will fail
[ 4852.716924] Lustre: Evicted from MGS (at 192.168.204.122@tcp) after server handle changed from 0x7693a547d124d1ce to 0x7693a547d12784f9
[ 5043.338698] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 20:38:35 (1786754315)
[ 5049.454370] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5065.715533] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 192.168.204.122@tcp) was lost; in progress operations using this service will fail
[ 5065.752946] Lustre: Evicted from MGS (at 192.168.204.122@tcp) after server handle changed from 0x7693a547d12784f9 to 0x7693a547d12c4eee
[ 5065.782654] Lustre: MGC192.168.204.122@tcp: Connection restored to 192.168.204.122@tcp (at 192.168.204.122@tcp)
[ 5065.794015] Lustre: Skipped 10 previous similar messages
[ 5067.772072] LustreError: lustre-MDT0000-mdc-ffff988207e4d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5090.492925] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 20:39:23 (1786754363)
[ 5091.398900] Lustre: Mounted lustre-client - version 2.17.57_2_g67a826c
[ 5099.351867] LustreError: lustre-OST0000-osc-ffff988207e4d800: operation ldlm_enqueue to node 192.168.204.122@tcp failed: rc = -107
[ 5099.373988] LustreError: lustre-OST0000-osc-ffff988207e4d800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5112.275660] Lustre: Unmounted lustre-client
[ 5119.631431] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 20:39:52 (1786754392)
[ 5127.035257] LustreError: lustre-MDT0000-mdc-ffff988207e4d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5127.047407] LustreError: 60916:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 5135.341359] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 20:40:08 (1786754408)
[ 5135.880354] LustreError: 61620:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 5140.959339] LustreError: 61620:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=0
[ 5140.973853] LustreError: 61620:0:(import.c:711:ptlrpc_connect_import_locked()) already connected
[ 5142.505050] LustreError: lustre-MDT0000-mdc-ffff988207e4d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5142.517510] LustreError: 61638:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 5142.579254] LustreError: 61643:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5143.643738] LustreError: 61649:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5143.666788] LustreError: 61649:0:(file.c:6156:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5145.977831] LustreError: 61671:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5145.997065] LustreError: 61671:0:(file.c:6156:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5150.847163] LustreError: 61716:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5150.866612] LustreError: 61716:0:(file.c:6156:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 5161.251600] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 20:40:34 (1786754434)
[ 5206.386812] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5208.079680] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5219.649795] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 20:41:32 (1786754492)
[ 5263.367466] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5265.103408] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5277.037556] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 20:42:29 (1786754549)
[ 5346.084478] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5348.118673] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5357.785926] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 20:43:50 (1786754630)
[ 5398.580796] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5400.154330] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5407.262801] Lustre: Unmounted lustre-client
[ 5467.858756] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5470.610194] Lustre: Mounted lustre-client - version 2.17.57_2_g67a826c
[ 5478.268843] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 20:45:51 (1786754751)
[ 5481.938258] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5484.331297] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 20:45:56 (1786754756)
[ 5521.540552] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 20:46:33 (1786754793)
[ 5523.141579] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5525.506578] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 20:46:37 (1786754797)
[ 5527.141793] Lustre: *** cfs_fail_loc=805, val=0***
[ 5527.230185] Lustre: Mounted lustre-client - version 2.17.57_2_g67a826c
[ 5533.453444] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5553.055186] Lustre: 2348:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786754811/real 1786754811]  req@ffff988210901c00 x1873542532568448/t0(0) o400->MGC192.168.204.122@tcp@192.168.204.122@tcp:26/25 lens 224/224 e 0 to 1 dl 1786754827 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5553.087776] Lustre: 2348:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5553.096608] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 192.168.204.122@tcp) was lost; in progress operations using this service will fail
[ 5553.113792] Lustre: lustre-MDT0000-mdc-ffff98821194d800: Connection to lustre-MDT0000 (at 192.168.204.122@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5553.122386] Lustre: Skipped 12 previous similar messages
[ 5563.385620] Lustre: Evicted from MGS (at 192.168.204.122@tcp) after server handle changed from 0x7693a547d12c5b66 to 0x7693a547d12c5f02
[ 5572.422887] Lustre: Unmounted lustre-client
[ 5582.347530] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 20:47:34 (1786754854)
[ 5624.451093] LustreError: 2346:0:(client.c:3435:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff98820410d880 x1873542532580352/t94489280517(94489280517) o101->lustre-MDT0000-mdc-ffff9882068ca000@192.168.204.122@tcp:12/10 lens 648/608 e 0 to 0 dl 1786754914 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5638.262984] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5639.946902] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5648.730514] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 20:48:41 (1786754921)
[ 5650.286993] LustreError: lustre-OST0000-osc-ffff9882068ca000: operation ost_write to node 192.168.204.122@tcp failed: rc = -107
[ 5650.305510] LustreError: Skipped 2 previous similar messages
[ 5650.324325] LustreError: lustre-OST0000-osc-ffff9882068ca000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5650.332115] Lustre: 2350:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.122@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5650.374613] LustreError: 72836:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff9882068ca000: namespace resource [0x240000400:0x3542:0x0].0x0 (ffff988204384000) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5659.518633] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 20:48:52 (1786754932)
[ 5660.991430] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5662.972738] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 20:48:55 (1786754935)
[ 5664.712428] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5666.474866] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 20:48:59 (1786754939)
[ 5668.374300] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5670.421978] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 20:49:02 (1786754942)
[ 5672.154625] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5674.140586] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 20:49:06 (1786754946)
[ 5675.715798] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5678.169469] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 20:49:10 (1786754950)
[ 5679.978484] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5682.288186] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 20:49:14 (1786754954)
[ 5684.280812] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5686.298560] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 20:49:18 (1786754958)
[ 5687.791943] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5689.682822] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 20:49:22 (1786754962)
[ 5691.858560] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5694.243911] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 20:49:26 (1786754966)
[ 5696.084088] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5698.100174] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 20:49:30 (1786754970)
[ 5699.770944] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5701.641332] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 20:49:34 (1786754974)
[ 5704.666938] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5706.693446] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 20:49:39 (1786754979)
[ 5721.607709] Lustre: MGC192.168.204.122@tcp: Connection restored to 192.168.204.122@tcp (at 192.168.204.122@tcp)
[ 5721.620609] Lustre: Skipped 15 previous similar messages
[ 5731.834355] Lustre: 68291:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.204.122@tcp: IR log lustre-cliir failed, not fatal: rc = -5
[ 5731.867724] LustreError: 77215:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.204.122@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (ffff988204384d00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5751.600739] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 20:50:24 (1786755024)
[ 5783.726290] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 20:50:56 (1786755056)
[ 5784.141192] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5795.939382] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 20:51:08 (1786755068)
[ 5796.838312] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5807.659157] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 20:51:20 (1786755080)
[ 5807.885532] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5817.805751] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 20:51:30 (1786755090)
[ 5818.298549] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5828.567751] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 20:51:41 (1786755101)
[ 5828.996968] Lustre: *** cfs_fail_loc=510, val=3***
[ 5838.787138] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 20:51:51 (1786755111)
[ 5839.146741] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5910.189514] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 20:53:02 (1786755182)
[ 5910.814385] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5977.921270] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 20:54:10 (1786755250)
[ 5978.569292] LustreError: 83058:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5981.466867] LustreError: lustre-MDT0000-mdc-ffff9882068ca000: operation ldlm_enqueue to node 192.168.204.122@tcp failed: rc = -107
[ 5981.485382] LustreError: lustre-MDT0000-mdc-ffff9882068ca000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5981.524911] LustreError: 83072:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9882068ca000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff98820325c600) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5982.663699] LustreError: 83058:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5984.818186] LustreError: 83088:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5987.874955] LustreError: lustre-MDT0000-mdc-ffff9882068ca000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5987.910466] LustreError: 83102:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9882068ca000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff98820650c100) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5987.939773] LustreError: 83102:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5988.905406] LustreError: 83088:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5995.962293] LustreError: lustre-MDT0000-mdc-ffff9882068ca000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5996.457262] LustreError: 83144:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5999.390737] LustreError: lustre-MDT0000-mdc-ffff9882068ca000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5999.490687] LustreError: 83163:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5999.508395] LustreError: 83163:0:(file.c:6156:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 6000.543502] LustreError: 83144:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 6000.562060] LustreError: 83144:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9882068ca000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 6000.608775] LustreError: 83169:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 6000.624495] LustreError: 83169:0:(file.c:6156:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 6002.956357] LustreError: 83191:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 6002.965484] LustreError: 83191:0:(file.c:6156:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 6006.590979] LustreError: 83218:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 6009.603352] LustreError: lustre-MDT0000-mdc-ffff9882068ca000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6009.636991] LustreError: 83233:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9882068ca000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff98820325c500) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6009.672156] LustreError: 83233:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 6010.671317] LustreError: 83218:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 6017.668970] LustreError: lustre-MDT0000-mdc-ffff9882068ca000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6017.682483] LustreError: 83261:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9882068ca000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff98820325c000) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6017.705481] LustreError: 83261:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 6018.124785] LustreError: 83274:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 6020.893702] LustreError: lustre-MDT0000-mdc-ffff9882068ca000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6022.199295] LustreError: 83274:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 6025.202393] LustreError: lustre-MDT0000-mdc-ffff9882068ca000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6025.290065] LustreError: 83321:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 6025.297118] LustreError: 83321:0:(file.c:6156:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 6026.411686] LustreError: 83301:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9882068ca000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 6034.528283] LustreError: 83373:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 6034.538134] LustreError: 83373:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 6037.738324] LustreError: lustre-MDT0000-mdc-ffff9882068ca000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6037.821732] LustreError: 83395:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 6037.833166] LustreError: 83395:0:(file.c:6156:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 6038.599173] LustreError: 83373:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 6038.605365] LustreError: 83373:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 6038.617600] LustreError: 83373:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9882068ca000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 6048.471783] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 20:55:21 (1786755321)
[ 6068.221882] LustreError: 79802:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 6068.229310] LustreError: 79802:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 6072.311875] LustreError: 79802:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 6072.326186] LustreError: 79802:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 6078.893298] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 20:55:51 (1786755351)
[ 6129.546686] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 20:56:42 (1786755402)
[ 6185.951452] Lustre: 85551:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786755405/real 1786755405]  req@ffff988210998380 x1873542532712704/t0(0) o101->lustre-MDT0000-mdc-ffff9882068ca000@192.168.204.122@tcp:12/10 lens 576/1152 e 0 to 1 dl 1786755460 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'stat.0' uid:0 gid:0 projid:0
[ 6185.986438] Lustre: 85551:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 20 previous similar messages
[ 6186.000542] Lustre: lustre-MDT0000-mdc-ffff9882068ca000: Connection to lustre-MDT0000 (at 192.168.204.122@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6186.017402] Lustre: Skipped 17 previous similar messages
[ 6193.718868] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 20:57:46 (1786755466)
[ 6226.708290] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 20:58:19 (1786755499)
[ 6227.257478] Lustre: Mounted lustre-client - version 2.17.57_2_g67a826c
[ 6351.138919] Lustre: Unmounted lustre-client
[ 6359.434551] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 21:00:31 (1786755631)
[ 6360.117286] LustreError: 87706:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 6363.101803] LustreError: lustre-OST0000-osc-ffff9882068ca000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6363.123640] LustreError: 87789:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff9882068ca000: namespace resource [0x240000400:0x356d:0x0].0x0 (ffff98820650c400) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6363.141127] LustreError: 87789:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 2 previous similar messages
[ 6363.577347] LustreError: 87706:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout interrupted
[ 6363.585387] Lustre: 2349:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.122@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 6371.093877] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 21:00:44 (1786755644)
[ 6429.682708] Lustre: lustre-MDT0000-mdc-ffff9882068ca000: Connection restored to 192.168.204.122@tcp (at 192.168.204.122@tcp)
[ 6429.691617] Lustre: Skipped 14 previous similar messages
[ 6437.699961] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 21:01:50 (1786755710)
[ 6439.212750] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 6441.230382] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 21:01:54 (1786755714)
[ 6504.696755] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6506.391609] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 21:02:59 (1786755779)
[ 6565.566610] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 21:03:57 (1786755837)
[ 6568.842169] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6570.705620] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 21:04:03 (1786755843)
[ 6572.323708] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6574.354981] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 21:04:07 (1786755847)
[ 6613.110565] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 21:04:45 (1786755885)
[ 6626.250668] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6654.943858] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 192.168.204.122@tcp) was lost; in progress operations using this service will fail
[ 6654.960095] LustreError: Skipped 3 previous similar messages
[ 6665.195112] Lustre: Evicted from MGS (at 192.168.204.122@tcp) after server handle changed from 0x7693a547d12c6989 to 0x7693a547d12c82db
[ 6665.198903] Lustre: Skipped 3 previous similar messages
[ 6679.182770] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6681.040671] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6691.368564] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 21:06:04 (1786755964)
[ 6694.120292] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6696.968182] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 21:06:09 (1786755969)
[ 6711.283590] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 192.168.204.122@tcp) was lost; in progress operations using this service will fail
[ 6711.309762] Lustre: Evicted from MGS (at 192.168.204.122@tcp) after server handle changed from 0x7693a547d12c82db to 0x7693a547d12c8615
[ 6726.730822] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 21:06:39 (1786755999)
[ 6776.485403] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 21:07:29 (1786756049)
[ 6784.016492] LustreError: lustre-OST0000-osc-ffff9882068ca000: operation ost_setattr to node 192.168.204.122@tcp failed: rc = -107
[ 6784.024478] LustreError: Skipped 11 previous similar messages
[ 6786.463252] Lustre: 2348:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786756005/real 1786756005]  req@ffff98820410f480 x1873542532812160/t0(0) o400->lustre-MDT0000-mdc-ffff9882068ca000@192.168.204.122@tcp:12/10 lens 224/224 e 0 to 1 dl 1786756060 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6786.493542] Lustre: 2348:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 13 previous similar messages
[ 6840.752690] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6846.311176] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6931.938094] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 192.168.204.122@tcp) was lost; in progress operations using this service will fail
[ 6931.963036] LustreError: Skipped 1 previous similar message
[ 6937.065382] Lustre: lustre-MDT0000-mdc-ffff9882068ca000: Connection to lustre-MDT0000 (at 192.168.204.122@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6937.082159] Lustre: Skipped 8 previous similar messages
[ 6942.317484] Lustre: Evicted from MGS (at 192.168.204.122@tcp) after server handle changed from 0x7693a547d12c894f to 0x7693a547d12c9ab3
[ 6942.340539] Lustre: Skipped 1 previous similar message
[ 6949.305256] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6950.989113] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6998.055743] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6999.907332] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 7142.045850] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 21:13:34 (1786756414)
[ 7210.895499] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 7216.233255] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 7487.705271] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 21:19:20 (1786756760)
[ 7670.245726] Lustre: lustre-MDT0000-mdc-ffff9882068ca000: Connection to lustre-MDT0000 (at 192.168.204.122@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7670.248297] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 192.168.204.122@tcp) was lost; in progress operations using this service will fail
[ 7670.260775] Lustre: Skipped 2 previous similar messages
[ 7670.273696] LustreError: Skipped 1 previous similar message
[ 7670.292683] Lustre: Evicted from MGS (at 192.168.204.122@tcp) after server handle changed from 0x7693a547d12c9c73 to 0x7693a547d13c3ef9
[ 7670.317263] Lustre: Skipped 1 previous similar message
[ 7670.337156] Lustre: MGC192.168.204.122@tcp: Connection restored to 192.168.204.122@tcp (at 192.168.204.122@tcp)
[ 7670.352725] Lustre: Skipped 12 previous similar messages
[ 7693.791186] Lustre: 2348:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786756929/real 1786756929]  req@ffff98820667e680 x1873542550459520/t0(0) o400->lustre-MDT0000-mdc-ffff9882068ca000@192.168.204.122@tcp:12/10 lens 224/224 e 0 to 1 dl 1786756967 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7693.829484] Lustre: 2348:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 17 previous similar messages
[ 7701.618984] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 21:22:54 (1786756974)
[ 7703.176186] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7704.567075] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 21:22:57 (1786756977)
[ 7706.747354] LustreError: lustre-MDT0000-mdc-ffff9882068ca000: operation ldlm_enqueue to node 192.168.204.122@tcp failed: rc = -107
[ 7706.762108] LustreError: Skipped 104 previous similar messages
[ 7706.781386] LustreError: lustre-MDT0000-mdc-ffff9882068ca000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7715.977408] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 21:23:08 (1786756988)
[ 7883.891272] LustreError: lustre-OST0000-osc-ffff9882068ca000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7885.871663] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 21:25:58 (1786757158)
[ 7911.401870] Lustre: lustre-OST0000-osc-ffff9882068ca000: Connection restored to 192.168.204.122@tcp (at 192.168.204.122@tcp)
[ 7911.420801] Lustre: Skipped 2 previous similar messages
[ 7930.625971] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 21:26:43 (1786757203)
[ 7932.138453] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7934.389843] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 21:26:46 (1786757206)
[ 7936.559570] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7939.364800] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 21:26:51 (1786757211)
[ 7976.653706] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 21:27:29 (1786757249)
[ 8016.911330] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 192.168.204.122@tcp) was lost; in progress operations using this service will fail
[ 8016.967983] Lustre: Evicted from MGS (at 192.168.204.122@tcp) after server handle changed from 0x7693a547d13c3ef9 to 0x7693a547d13c6430
[ 8016.989938] Lustre: Skipped 1 previous similar message
[ 8040.350550] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 21:28:32 (1786757312)
[ 8042.493510] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 8045.058041] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 21:28:37 (1786757317)
[ 8046.884802] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 8048.841663] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 21:28:41 (1786757321)
[ 8064.983971] Lustre: Unmounted lustre-client
[ 8066.861455] Lustre: Mounted lustre-client - version 2.17.57_2_g67a826c
[ 8070.150653] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 8092.648622] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 192.168.204.122@tcp) was lost; in progress operations using this service will fail
[ 8092.703267] Lustre: MGC192.168.204.122@tcp: Connection restored to 192.168.204.122@tcp (at 192.168.204.122@tcp)
[ 8092.709546] Lustre: Skipped 2 previous similar messages
[ 8094.195829] Lustre: 106846:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.204.122@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 8114.959889] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2