[ 2171.097107] LustreError: lustre-OST0000-osc-ffff96a50371c000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2171.120417] Lustre: 2395:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.104@tcp:/lustre/fid: [0x200000405:0x2e:0x0]// may get corrupted (rc -5)
[ 2171.143180] Lustre: 2394:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.104@tcp:/lustre/fid: [0x200000405:0x2f:0x0]// may get corrupted (rc -108)
[ 2180.358637] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 11:04:06 (1785942246)
[ 2184.310548] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 2186.967470] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 11:04:12 (1785942252)
[ 2189.101676] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 2191.374881] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 11:04:17 (1785942257)
[ 2196.004937] LustreError: lustre-MDT0000-mdc-ffff96a50371c000: operation mds_close to node 192.168.203.104@tcp failed: rc = -19
[ 2196.016700] LustreError: Skipped 2 previous similar messages
[ 2213.344075] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 192.168.203.104@tcp) was lost; in progress operations using this service will fail
[ 2223.608701] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0xcc06afcff842e86e to 0xcc06afcff842fec5
[ 2225.075607] Lustre: 16844:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.104@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2336.682706] LustreError: lustre-MDT0000-mdc-ffff96a50371c000: operation mds_reint to node 192.168.203.104@tcp failed: rc = -19
[ 2356.705151] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 192.168.203.104@tcp) was lost; in progress operations using this service will fail
[ 2365.949067] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0xcc06afcff842fec5 to 0xcc06afcff844cc0f
[ 2365.989424] Lustre: MGC192.168.203.104@tcp: Connection restored to 192.168.203.104@tcp (at 192.168.203.104@tcp)
[ 2366.010262] Lustre: Skipped 12 previous similar messages
[ 2386.536192] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 11:07:32 (1785942452)
[ 2387.210579] Lustre: *** cfs_fail_loc=305, val=0***
[ 2387.212570] Lustre: Skipped 2 previous similar messages
[ 2414.056641] Lustre: lustre-MDT0000-mdc-ffff96a50371c000: Connection to lustre-MDT0000 (at 192.168.203.104@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2414.075544] Lustre: Skipped 12 previous similar messages
[ 2435.551240] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 192.168.203.104@tcp) was lost; in progress operations using this service will fail
[ 2444.894893] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0xcc06afcff844cc0f to 0xcc06afcff844e77c
[ 2463.710331] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2466.239854] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2477.524694] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 11:09:03 (1785942543)
[ 2498.015720] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 192.168.203.104@tcp) was lost; in progress operations using this service will fail
[ 2498.052807] LustreError: lustre-MDT0000-mdc-ffff96a50371c000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2498.075584] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0xcc06afcff844e77c to 0xcc06afcff844efb0
[ 2533.493308] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 11:09:59 (1785942599)
[ 2552.598067] LustreError: lustre-OST0000-osc-ffff96a50371c000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2589.801868] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 11:10:55 (1785942655)
[ 2621.855120] Lustre: 2397:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785942673/real 1785942673]  req@ffff96a504991180 x1872693942281216/t0(0) o400->MGC192.168.203.104@tcp@192.168.203.104@tcp:26/25 lens 224/224 e 0 to 1 dl 1785942689 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2621.924103] Lustre: 2397:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 10 previous similar messages
[ 2621.949958] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 192.168.203.104@tcp) was lost; in progress operations using this service will fail
[ 2632.204286] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0xcc06afcff844efb0 to 0xcc06afcff845b84f
[ 2659.962871] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2663.021252] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2749.730119] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 192.168.203.104@tcp) was lost; in progress operations using this service will fail
[ 2774.676409] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0xcc06afcff845b84f to 0xcc06afcff8482a82
[ 2784.578804] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2787.669569] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2873.125301] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 192.168.203.104@tcp) was lost; in progress operations using this service will fail
[ 2883.603779] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0xcc06afcff8482a82 to 0xcc06afcff84a200c
[ 2910.421870] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2913.303741] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2944.520507] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 11:16:50 (1785943010)
[ 2951.133389] LustreError: lustre-MDT0000-mdc-ffff96a50371c000: operation mds_reint to node 192.168.203.104@tcp failed: rc = -19
[ 2970.600200] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 192.168.203.104@tcp) was lost; in progress operations using this service will fail
[ 2980.894314] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0xcc06afcff84a200c to 0xcc06afcff84b2cba
[ 2980.920107] Lustre: MGC192.168.203.104@tcp: Connection restored to 192.168.203.104@tcp (at 192.168.203.104@tcp)
[ 2980.934757] Lustre: Skipped 11 previous similar messages
[ 2993.141216] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 3026.411805] Lustre: 16844:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.104@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 3040.150882] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 3048.273338] Lustre: lustre-MDT0000-mdc-ffff96a50371c000: Connection to lustre-MDT0000 (at 192.168.203.104@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3048.299491] Lustre: Skipped 7 previous similar messages
[ 3088.633188] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 3123.681497] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 192.168.203.104@tcp) was lost; in progress operations using this service will fail
[ 3123.699502] LustreError: Skipped 2 previous similar messages
[ 3132.921675] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0xcc06afcff84bc2ae to 0xcc06afcff84c3715
[ 3132.939372] Lustre: Skipped 2 previous similar messages
[ 3134.774964] Lustre: 16844:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.104@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 3148.010837] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 3215.910162] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 3246.940850] LustreError: lustre-MDT0000-mdc-ffff96a50371c000: operation mds_close to node 192.168.203.104@tcp failed: rc = -19
[ 3246.944775] LustreError: Skipped 4 previous similar messages
[ 3267.808517] Lustre: 2395:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785943319/real 1785943319]  req@ffff96a510f5c380 x1872693949024128/t0(0) o400->MGC192.168.203.104@tcp@192.168.203.104@tcp:26/25 lens 224/224 e 0 to 1 dl 1785943335 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3267.862050] Lustre: 2395:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
[ 3290.518772] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 3393.857333] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 11:24:19 (1785943459)
[ 3454.339761] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3457.693763] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3740.567281] Lustre: lustre-OST0000-osc-ffff96a50371c000: Connection to lustre-OST0000 (at 192.168.203.104@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3740.587386] Lustre: Skipped 5 previous similar messages
[ 3765.746553] Lustre: lustre-OST0000-osc-ffff96a50371c000: Connection restored to 192.168.203.104@tcp (at 192.168.203.104@tcp)
[ 3765.767057] Lustre: Skipped 14 previous similar messages
[ 3787.171362] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3790.444649] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4072.407954] LustreError: lustre-OST0000-osc-ffff96a50371c000: operation ldlm_enqueue to node 192.168.203.104@tcp failed: rc = -107
[ 4072.424273] LustreError: Skipped 4 previous similar messages
[ 4121.060316] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4123.572673] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4371.050236] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 11:40:36 (1785944436)
[ 4388.831188] Lustre: 51963:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785944441/real 1785944441]  req@ffff96a520e29f80 x1872693965364992/t0(0) o101->lustre-MDT0000-mdc-ffff96a50371c000@192.168.203.104@tcp:12/10 lens 576/1152 e 0 to 1 dl 1785944457 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 4388.878874] Lustre: 51963:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 4388.894405] Lustre: lustre-MDT0000-mdc-ffff96a50371c000: Connection to lustre-MDT0000 (at 192.168.203.104@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4388.922463] Lustre: Skipped 1 previous similar message
[ 4388.975725] Lustre: lustre-MDT0000-mdc-ffff96a50371c000: Connection restored to 192.168.203.104@tcp (at 192.168.203.104@tcp)
[ 4388.990449] Lustre: Skipped 1 previous similar message
[ 4402.355646] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 11:41:09 (1785944469)
[ 4429.284116] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 11:41:35 (1785944495)
[ 4458.824765] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 11:42:04 (1785944524)
[ 4459.740176] Lustre: Mounted lustre-client - version 2.17.56_50_g82e980f
[ 4491.743335] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 192.168.203.104@tcp) was lost; in progress operations using this service will fail
[ 4491.752955] LustreError: Skipped 3 previous similar messages
[ 4501.010698] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0xcc06afcff84e34ef to 0xcc06afcff8618c7a
[ 4501.028302] Lustre: Skipped 3 previous similar messages
[ 4525.763388] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4527.618677] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4529.913559] Lustre: Unmounted lustre-client
[ 4538.064387] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 11:43:24 (1785944604)
[ 4647.271381] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 11:45:13 (1785944713)
[ 4698.609740] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 11:46:04 (1785944764)
[ 4702.176300] Lustre: Unmounted lustre-client
[ 4743.853404] Lustre: Mounted lustre-client - version 2.17.56_50_g82e980f
[ 4754.153778] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 11:47:00 (1785944820)
[ 4754.384885] LustreError: 57838:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4755.443404] LustreError: 57838:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4755.558905] Lustre: *** cfs_fail_loc=305, val=0***
[ 4779.580924] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 11:47:26 (1785944846)
[ 4780.094256] Lustre: Mounted lustre-client - version 2.17.56_50_g82e980f
[ 4783.858609] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4794.237128] Lustre: Unmounted lustre-client
[ 4802.586644] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 11:47:49 (1785944869)
[ 4908.657787] LustreError: lustre-MDT0000-mdc-ffff96a502be8000: operation mds_reint to node 192.168.203.104@tcp failed: rc = -107
[ 4928.495768] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 192.168.203.104@tcp) was lost; in progress operations using this service will fail
[ 4928.545201] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0xcc06afcff8619e5c to 0xcc06afcff8644f8f
[ 4930.408311] Lustre: 57230:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.104@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 5108.040705] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 11:52:53 (1785945173)
[ 5122.770633] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5133.297939] Lustre: lustre-MDT0000-mdc-ffff96a502be8000: Connection to lustre-MDT0000 (at 192.168.203.104@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5133.314849] Lustre: Skipped 12 previous similar messages
[ 5148.665647] LustreError: lustre-MDT0000-mdc-ffff96a502be8000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5148.668620] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 192.168.203.104@tcp) was lost; in progress operations using this service will fail
[ 5148.714445] Lustre: lustre-MDT0000-mdc-ffff96a502be8000: Connection restored to 192.168.203.104@tcp (at 192.168.203.104@tcp)
[ 5148.733251] Lustre: Skipped 14 previous similar messages
[ 5148.745208] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0xcc06afcff8644f8f to 0xcc06afcff86811f3
[ 5173.977580] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 11:54:00 (1785945240)
[ 5174.555631] Lustre: Mounted lustre-client - version 2.17.56_50_g82e980f
[ 5182.162287] LustreError: lustre-OST0000-osc-ffff96a502be8000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5192.671778] Lustre: 2395:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785945244/real 1785945244]  req@ffff96a518d03b80 x1872693969733632/t0(0) o103->lustre-OST0000-osc-ffff96a520528000@192.168.203.104@tcp:17/18 lens 328/224 e 0 to 1 dl 1785945260 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295
[ 5192.711153] Lustre: 2395:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 36 previous similar messages
[ 5197.425878] Lustre: Unmounted lustre-client
[ 5207.706215] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 11:54:33 (1785945273)
[ 5215.388720] LustreError: lustre-MDT0000-mdc-ffff96a502be8000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5215.408534] LustreError: 62550:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 5227.686988] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 11:54:54 (1785945294)
[ 5228.320035] LustreError: 63263:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 5232.505917] LustreError: lustre-MDT0000-mdc-ffff96a502be8000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5232.518288] LustreError: 63279:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 5232.528608] LustreError: 63263:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=803
[ 5232.531430] LustreError: 63263:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting
[ 5232.583805] LustreError: 63284:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5233.730210] LustreError: 63291:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5233.748567] LustreError: 63291:0:(file.c:6154:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5236.223372] LustreError: 63313:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5236.235900] LustreError: 63313:0:(file.c:6154:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5240.991742] LustreError: 63358:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5241.003496] LustreError: 63358:0:(file.c:6154:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 5253.012646] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 11:55:19 (1785945319)
[ 5308.536450] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5311.552946] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5326.131263] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 11:56:32 (1785945392)
[ 5377.676969] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5379.676408] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5395.371789] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 11:57:41 (1785945461)
[ 5472.048923] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5474.324976] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5488.764735] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 11:59:14 (1785945554)
[ 5540.109891] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5541.913632] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5549.808730] Lustre: Unmounted lustre-client
[ 5625.947561] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5628.919884] Lustre: Mounted lustre-client - version 2.17.56_50_g82e980f
[ 5636.815983] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 12:01:43 (1785945703)
[ 5639.932747] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5641.989971] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 12:01:48 (1785945708)
[ 5685.523974] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 12:02:32 (1785945752)
[ 5687.500925] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5689.252615] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 12:02:36 (1785945756)
[ 5691.280401] Lustre: *** cfs_fail_loc=805, val=0***
[ 5691.385942] Lustre: Mounted lustre-client - version 2.17.56_50_g82e980f
[ 5703.274741] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5722.079729] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 192.168.203.104@tcp) was lost; in progress operations using this service will fail
[ 5747.708833] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0xcc06afcff8681d6f to 0xcc06afcff8681fc2
[ 5751.794681] Lustre: lustre-MDT0000-mdc-ffff96a502e29000: Connection restored to 192.168.203.104@tcp (at 192.168.203.104@tcp)
[ 5751.812700] Lustre: Skipped 8 previous similar messages
[ 5767.026677] Lustre: Unmounted lustre-client
[ 5780.305867] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 12:04:05 (1785945845)
[ 5792.741899] Lustre: lustre-MDT0000-mdc-ffff96a503718800: Connection to lustre-MDT0000 (at 192.168.203.104@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5792.768372] Lustre: Skipped 10 previous similar messages
[ 5797.857368] Lustre: 73227:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785945850/real 1785945850]  req@ffff96a518c3dc00 x1872693969843072/t0(0) o36->lustre-MDT0000-mdc-ffff96a503718800@192.168.203.104@tcp:12/10 lens 552/576 e 0 to 1 dl 1785945866 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'mkdir.0' uid:0 gid:0 projid:4294967295
[ 5797.907355] Lustre: 73227:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5818.406346] LustreError: 2393:0:(client.c:3435:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff96a518c3df80 x1872693969843328/t94489280517(94489280517) o101->lustre-MDT0000-mdc-ffff96a503718800@192.168.203.104@tcp:12/10 lens 520/664 e 0 to 0 dl 1785945902 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5838.374860] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5842.394213] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5857.289560] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 12:05:23 (1785945923)
[ 5864.444222] LustreError: 2393:0:(osc_request.c:1024:osc_init_grant()) lustre-OST0000-osc-ffff96a503718800: granted 8437760 but already consumed 16850944
[ 5864.453442] LustreError: lustre-OST0000-osc-ffff96a503718800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5864.468902] Lustre: 2396:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.104@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5864.757037] LustreError: 74619:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff96a503718800: namespace resource [0x280000401:0x30c2:0x0].0x0 (ffff96a505e5d600) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5874.635441] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 12:05:40 (1785945940)
[ 5908.254552] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 12:06:13 (1785945973)
[ 5936.059455] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 12:06:42 (1785946002)
[ 5964.516437] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 12:07:10 (1785946030)
[ 6035.215660] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 12:08:21 (1785946101)
[ 6105.648330] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 12:09:31 (1785946171)
[ 6133.917482] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 12:09:59 (1785946199)
[ 6203.436440] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 12:11:09 (1785946269)
[ 6232.316689] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 12:11:38 (1785946298)
[ 6265.429224] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 12:12:10 (1785946330)
[ 6292.333240] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 12:12:39 (1785946359)
[ 6292.779644] Lustre: Unmounted lustre-client
[ 6346.888405] Lustre: Mounted lustre-client - version 2.17.56_50_g82e980f
[ 6369.262447] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 12:13:54 (1785946434)
[ 6400.990398] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 12:14:27 (1785946467)
[ 6408.693774] Lustre: lustre-MDT0000-mdc-ffff96a520bef000: Connection to lustre-MDT0000 (at 192.168.203.104@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6408.716038] Lustre: Skipped 6 previous similar messages
[ 6424.052571] Lustre: 2394:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785946476/real 1785946476]  req@ffff96a518c3c700 x1872693969979904/t0(0) o400->MGC192.168.203.104@tcp@192.168.203.104@tcp:26/25 lens 224/224 e 0 to 1 dl 1785946492 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6424.100067] Lustre: 2394:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
[ 6424.126490] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 192.168.203.104@tcp) was lost; in progress operations using this service will fail
[ 6424.158214] LustreError: Skipped 1 previous similar message
[ 6431.797990] Lustre: lustre-MDT0000-mdc-ffff96a520bef000: Connection restored to 192.168.203.104@tcp (at 192.168.203.104@tcp)
[ 6431.815757] Lustre: Skipped 8 previous similar messages
[ 6433.923804] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0xcc06afcff86844c8 to 0xcc06afcff8684beb
[ 6433.934825] Lustre: Skipped 1 previous similar message
[ 6447.946817] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 12:15:13 (1785946513)
[ 6482.538566] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 12:15:49 (1785946549)
[ 6483.398913] Lustre: *** cfs_fail_loc=51b, val=3***
[ 6496.552842] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 12:16:02 (1785946562)
[ 6497.542229] Lustre: *** cfs_fail_loc=51b, val=4***
[ 6512.064935] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 12:16:18 (1785946578)
[ 6512.485880] Lustre: *** cfs_fail_loc=50f, val=3***
[ 6524.468455] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 12:16:30 (1785946590)
[ 6524.820314] Lustre: *** cfs_fail_loc=50f, val=4***
[ 6534.937880] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 12:16:41 (1785946601)
[ 6535.567357] Lustre: *** cfs_fail_loc=510, val=3***
[ 6545.997297] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 12:16:52 (1785946612)
[ 6546.457629] Lustre: *** cfs_fail_loc=51b, val=3***
[ 6558.184562] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 12:17:05 (1785946625)
[ 6558.567752] Lustre: *** cfs_fail_loc=51c, val=3***
[ 6624.885655] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 12:18:12 (1785946692)
[ 6625.394237] LustreError: 89430:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 6628.353077] LustreError: lustre-MDT0000-mdc-ffff96a520bef000: operation ldlm_enqueue to node 192.168.203.104@tcp failed: rc = -107
[ 6628.370371] LustreError: Skipped 3 previous similar messages
[ 6628.391026] LustreError: lustre-MDT0000-mdc-ffff96a520bef000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6628.410365] LustreError: 89444:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff96a520bef000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff96a5185bb200) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6629.479114] LustreError: 89430:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 6631.620230] LustreError: 89460:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 6634.657052] LustreError: lustre-MDT0000-mdc-ffff96a520bef000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6634.688424] LustreError: 89474:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff96a520bef000: namespace resource [0x20000afe2:0xc:0x0].0xc (ffff96a5185bb700) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6634.736359] LustreError: 89474:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 2 previous similar messages
[ 6634.743377] LustreError: 89479:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 6634.761907] LustreError: 89479:0:(file.c:6154:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 6635.706820] LustreError: 89460:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 6642.807467] LustreError: lustre-MDT0000-mdc-ffff96a520bef000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6643.362734] LustreError: 89516:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 6646.484129] LustreError: lustre-MDT0000-mdc-ffff96a520bef000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6646.577941] LustreError: 89536:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 6647.455123] LustreError: 89516:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 6647.478781] LustreError: 89516:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff96a520bef000: inode [0x20000afe2:0xc:0x0] mdc close failed: rc = -108
[ 6648.885543] LustreError: 89553:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 6648.901889] LustreError: 89553:0:(file.c:6154:ll_inode_revalidate_fini()) Skipped 12 previous similar messages
[ 6653.917219] LustreError: 89591:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 6657.991185] LustreError: 89591:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 6658.849416] LustreError: lustre-MDT0000-mdc-ffff96a520bef000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6665.798119] LustreError: lustre-MDT0000-mdc-ffff96a520bef000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6666.698364] LustreError: 89647:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 6670.799228] LustreError: 89647:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 6670.843236] LustreError: lustre-MDT0000-mdc-ffff96a520bef000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6670.894395] LustreError: 89647:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff96a520bef000: inode [0x20000afe2:0xc:0x0] mdc close failed: rc = -5
[ 6675.078889] Lustre: DEBUG MARKER: recovery-small test_120: @@@@@@ FAIL: DEADLOCK race failed