[ 2256.690158] LustreError: lustre-OST0000-osc-ffff91a8182b0800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2256.704037] Lustre: 2400:0:(llite_lib.c:4158:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.103@tcp:/lustre/fid: [0x240000402:0x9:0x0]// may get corrupted (rc -108)
[ 2266.588156] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 17:45:31 (1773611131)
[ 2269.508928] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 2271.699847] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 17:45:36 (1773611136)
[ 2274.407659] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 2276.823642] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 17:45:41 (1773611141)
[ 2281.517581] LustreError: lustre-MDT0000-mdc-ffff91a8182b0800: operation ldlm_enqueue to node 192.168.201.103@tcp failed: rc = -107
[ 2281.532813] LustreError: Skipped 2 previous similar messages
[ 2301.410207] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 192.168.201.103@tcp) was lost; in progress operations using this service will fail
[ 2311.678187] Lustre: Evicted from MGS (at 192.168.201.103@tcp) after server handle changed from 0xf65161857731eb7 to 0xf651618577338a3
[ 2416.754505] LustreError: lustre-MDT0000-mdc-ffff91a8182b0800: operation ldlm_enqueue to node 192.168.201.103@tcp failed: rc = -19
[ 2416.780433] LustreError: Skipped 1 previous similar message
[ 2434.527282] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 192.168.201.103@tcp) was lost; in progress operations using this service will fail
[ 2444.784559] Lustre: Evicted from MGS (at 192.168.201.103@tcp) after server handle changed from 0xf651618577338a3 to 0xf6516185774ce2c
[ 2444.817430] Lustre: MGC192.168.201.103@tcp: Connection restored to 192.168.201.103@tcp (at 192.168.201.103@tcp)
[ 2444.842706] Lustre: Skipped 12 previous similar messages
[ 2464.166355] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 17:48:48 (1773611328)
[ 2464.631772] Lustre: *** cfs_fail_loc=305, val=0***
[ 2464.634158] Lustre: Skipped 2 previous similar messages
[ 2492.913137] Lustre: lustre-MDT0000-mdc-ffff91a8182b0800: Connection to lustre-MDT0000 (at 192.168.201.103@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2492.932202] Lustre: Skipped 12 previous similar messages
[ 2509.281041] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 192.168.201.103@tcp) was lost; in progress operations using this service will fail
[ 2519.540617] Lustre: Evicted from MGS (at 192.168.201.103@tcp) after server handle changed from 0xf6516185774ce2c to 0xf6516185774ea95
[ 2536.118061] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2538.576887] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2551.922297] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 17:50:16 (1773611416)
[ 2575.876426] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 192.168.201.103@tcp) was lost; in progress operations using this service will fail
[ 2575.907578] Lustre: Evicted from MGS (at 192.168.201.103@tcp) after server handle changed from 0xf6516185774ea95 to 0xf6516185774ef18
[ 2577.985031] LustreError: lustre-MDT0000-mdc-ffff91a8182b0800: operation ldlm_enqueue to node 192.168.201.103@tcp failed: rc = -107
[ 2577.993691] LustreError: Skipped 2 previous similar messages
[ 2581.054360] LustreError: lustre-MDT0000-mdc-ffff91a8182b0800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2625.818576] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 17:51:30 (1773611490)
[ 2647.563250] LustreError: lustre-OST0000-osc-ffff91a8182b0800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2686.423844] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 17:52:31 (1773611551)
[ 2723.295776] Lustre: 2400:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773611573/real 1773611573]  req@ffff91a807a87800 x1859763768943488/t0(0) o400->MGC192.168.201.103@tcp@192.168.201.103@tcp:26/25 lens 224/224 e 0 to 1 dl 1773611589 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2723.373819] Lustre: 2400:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
[ 2723.397884] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 192.168.201.103@tcp) was lost; in progress operations using this service will fail
[ 2734.076141] Lustre: Evicted from MGS (at 192.168.201.103@tcp) after server handle changed from 0xf6516185774ef18 to 0xf65161857752fa0
[ 2756.785281] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2760.947024] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2848.482717] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 192.168.201.103@tcp) was lost; in progress operations using this service will fail
[ 2859.005758] Lustre: Evicted from MGS (at 192.168.201.103@tcp) after server handle changed from 0xf65161857752fa0 to 0xf65161857769fe4
[ 2878.292890] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2881.924254] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2949.756721] LustreError: lustre-MDT0000-mdc-ffff91a8182b0800: operation ldlm_enqueue to node 192.168.201.103@tcp failed: rc = -19
[ 2949.772447] LustreError: Skipped 3 previous similar messages
[ 2967.084918] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 192.168.201.103@tcp) was lost; in progress operations using this service will fail
[ 2977.262706] Lustre: Evicted from MGS (at 192.168.201.103@tcp) after server handle changed from 0xf65161857769fe4 to 0xf6516185778022f
[ 3003.258082] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3006.968803] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3039.465053] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 17:58:24 (1773611904)
[ 3066.338283] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 192.168.201.103@tcp) was lost; in progress operations using this service will fail
[ 3076.595932] Lustre: Evicted from MGS (at 192.168.201.103@tcp) after server handle changed from 0xf6516185778022f to 0xf6516185778c644
[ 3076.643663] Lustre: MGC192.168.201.103@tcp: Connection restored to 192.168.201.103@tcp (at 192.168.201.103@tcp)
[ 3076.682972] Lustre: Skipped 12 previous similar messages
[ 3091.981976] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 3096.231967] Lustre: lustre-MDT0000-mdc-ffff91a8182b0800: Connection to lustre-MDT0000 (at 192.168.201.103@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3096.260365] Lustre: Skipped 6 previous similar messages
[ 3137.337121] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 3191.742070] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 3226.016208] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 192.168.201.103@tcp) was lost; in progress operations using this service will fail
[ 3226.055765] LustreError: Skipped 2 previous similar messages
[ 3236.334015] Lustre: Evicted from MGS (at 192.168.201.103@tcp) after server handle changed from 0xf6516185779443d to 0xf6516185779a61a
[ 3236.343238] Lustre: Skipped 2 previous similar messages
[ 3245.999087] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 3311.700374] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 3360.735483] Lustre: 2401:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773612211/real 1773612211]  req@ffff91a807d34700 x1859763773609984/t0(0) o400->MGC192.168.201.103@tcp@192.168.201.103@tcp:26/25 lens 224/224 e 0 to 1 dl 1773612227 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3360.760311] Lustre: 2401:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
[ 3381.736645] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 3483.731538] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 18:05:48 (1773612348)
[ 3499.406323] LustreError: lustre-OST0000-osc-ffff91a8182b0800: operation ldlm_enqueue to node 192.168.201.103@tcp failed: rc = -107
[ 3499.429509] LustreError: Skipped 11 previous similar messages
[ 3543.489757] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3546.661591] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3829.577188] Lustre: lustre-OST0000-osc-ffff91a8182b0800: Connection to lustre-OST0000 (at 192.168.201.103@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3829.591604] Lustre: Skipped 6 previous similar messages
[ 3853.448725] Lustre: lustre-OST0000-osc-ffff91a8182b0800: Connection restored to 192.168.201.103@tcp (at 192.168.201.103@tcp)
[ 3853.464469] Lustre: Skipped 14 previous similar messages
[ 3873.627717] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3877.113534] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4160.685376] LustreError: lustre-OST0000-osc-ffff91a8182b0800: operation ldlm_enqueue to node 192.168.201.103@tcp failed: rc = -107
[ 4160.693509] LustreError: Skipped 1 previous similar message
[ 4201.626364] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4203.715814] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4456.514475] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 18:22:01 (1773613321)
[ 4473.312038] Lustre: 51919:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773613324/real 1773613324]  req@ffff91a807e1e680 x1859763790803584/t0(0) o101->lustre-MDT0000-mdc-ffff91a8182b0800@192.168.201.103@tcp:12/10 lens 576/1152 e 0 to 1 dl 1773613340 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 4473.349566] Lustre: 51919:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 4473.358667] Lustre: lustre-MDT0000-mdc-ffff91a8182b0800: Connection to lustre-MDT0000 (at 192.168.201.103@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4473.378815] Lustre: Skipped 1 previous similar message
[ 4473.421973] Lustre: lustre-MDT0000-mdc-ffff91a8182b0800: Connection restored to 192.168.201.103@tcp (at 192.168.201.103@tcp)
[ 4473.432370] Lustre: Skipped 1 previous similar message
[ 4480.912641] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 18:22:26 (1773613346)
[ 4518.047156] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 18:23:02 (1773613382)
[ 4553.815459] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 18:23:37 (1773613417)
[ 4555.249079] Lustre: Mounted lustre-client
[ 4592.097198] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 192.168.201.103@tcp) was lost; in progress operations using this service will fail
[ 4592.119651] LustreError: Skipped 3 previous similar messages
[ 4602.394931] Lustre: Evicted from MGS (at 192.168.201.103@tcp) after server handle changed from 0xf651618577bac98 to 0xf651618578ff10b
[ 4602.411129] Lustre: Skipped 3 previous similar messages
[ 4620.632663] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4622.785688] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4625.554719] LustreError: 54741:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff91a807fbd800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 4625.580681] LustreError: 54741:0:(lov_obd.c:783:lov_cleanup()) Skipped 1 previous similar message
[ 4625.723496] Lustre: Unmounted lustre-client
[ 4636.051937] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 18:25:00 (1773613500)
[ 4745.361075] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 18:26:49 (1773613609)
[ 4797.334269] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 18:27:42 (1773613662)
[ 4803.457203] LustreError: 56701:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff91a8182b0800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 4803.483980] LustreError: 56701:0:(lov_obd.c:783:lov_cleanup()) Skipped 1 previous similar message
[ 4803.674185] Lustre: Unmounted lustre-client
[ 4847.577905] Lustre: Mounted lustre-client
[ 4857.928273] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 18:28:41 (1773613721)
[ 4858.436875] LustreError: 57822:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4859.503172] LustreError: 57822:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4859.782883] Lustre: *** cfs_fail_loc=305, val=0***
[ 4889.466631] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 18:29:13 (1773613753)
[ 4890.375759] Lustre: Mounted lustre-client
[ 4894.330218] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4904.726178] Lustre: Unmounted lustre-client
[ 4916.180532] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 18:29:40 (1773613780)
[ 5037.685503] LustreError: lustre-MDT0000-mdc-ffff91a8066fa800: operation mds_reint to node 192.168.201.103@tcp failed: rc = -19
[ 5055.460133] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 192.168.201.103@tcp) was lost; in progress operations using this service will fail
[ 5066.753606] Lustre: Evicted from MGS (at 192.168.201.103@tcp) after server handle changed from 0xf651618578ffc2c to 0xf6516185792ab83
[ 5075.543722] Lustre: lustre-MDT0000-mdc-ffff91a8066fa800: Connection restored to 192.168.201.103@tcp (at 192.168.201.103@tcp)
[ 5075.608553] Lustre: Skipped 13 previous similar messages
[ 5267.415800] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 18:35:31 (1773614131)
[ 5282.867135] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5292.047286] Lustre: lustre-MDT0000-mdc-ffff91a8066fa800: Connection to lustre-MDT0000 (at 192.168.201.103@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5292.086549] Lustre: Skipped 12 previous similar messages
[ 5307.361339] Lustre: 2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773614158/real 1773614158]  req@ffff91a8034b0700 x1859763795160960/t0(0) o400->MGC192.168.201.103@tcp@192.168.201.103@tcp:26/25 lens 224/224 e 0 to 1 dl 1773614174 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5307.396391] Lustre: 2402:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 36 previous similar messages
[ 5307.410878] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 192.168.201.103@tcp) was lost; in progress operations using this service will fail
[ 5317.616664] Lustre: Evicted from MGS (at 192.168.201.103@tcp) after server handle changed from 0xf6516185792ab83 to 0xf65161857967550
[ 5322.758955] LustreError: lustre-MDT0000-mdc-ffff91a8066fa800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5353.593349] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 18:36:57 (1773614217)
[ 5354.545805] Lustre: Mounted lustre-client
[ 5362.930929] LustreError: lustre-OST0000-osc-ffff91a8066fa800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5376.830651] LustreError: 61918:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff91a807e3f800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 5376.845916] LustreError: 61918:0:(lov_obd.c:783:lov_cleanup()) Skipped 1 previous similar message
[ 5376.952953] Lustre: Unmounted lustre-client
[ 5390.306107] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 18:37:34 (1773614254)
[ 5399.105866] LustreError: lustre-MDT0000-mdc-ffff91a8066fa800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5399.133029] LustreError: 62581:0:(file.c:6089:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 5410.714646] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 18:37:55 (1773614275)
[ 5410.928753] LustreError: 63279:0:(recover.c:330:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 5415.345788] LustreError: lustre-MDT0000-mdc-ffff91a8066fa800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5415.388779] LustreError: 63302:0:(import.c:291:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 5415.418264] LustreError: 63279:0:(recover.c:330:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=521
[ 5415.433974] LustreError: 63279:0:(import.c:716:ptlrpc_connect_import_locked()) already connecting
[ 5415.508111] LustreError: 63307:0:(file.c:6089:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5416.667239] LustreError: 63313:0:(file.c:6089:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5416.682242] LustreError: 63313:0:(file.c:6089:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5419.325051] LustreError: 63336:0:(file.c:6089:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5419.340815] LustreError: 63336:0:(file.c:6089:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5424.298704] LustreError: 63382:0:(file.c:6089:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5424.321399] LustreError: 63382:0:(file.c:6089:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 5438.165441] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 18:38:22 (1773614302)
[ 5492.598279] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5495.112897] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5512.719229] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 18:39:36 (1773614376)
[ 5566.796651] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5569.519785] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5585.585352] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 18:40:49 (1773614449)
[ 5665.486730] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5668.513183] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5684.412817] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 18:42:27 (1773614547)
[ 5736.187390] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5738.958904] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5748.977326] LustreError: 69029:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff91a8066fa800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 5748.993469] LustreError: 69029:0:(lov_obd.c:783:lov_cleanup()) Skipped 1 previous similar message
[ 5749.121612] Lustre: Unmounted lustre-client
[ 5824.432846] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5828.459498] Lustre: Mounted lustre-client
[ 5838.903499] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 18:45:03 (1773614703)
[ 5842.236652] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5844.856088] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 18:45:09 (1773614709)
[ 5894.865089] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 18:45:58 (1773614758)
[ 5898.014169] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5900.620873] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 18:46:05 (1773614765)
[ 5903.434647] Lustre: *** cfs_fail_loc=805, val=0***
[ 5903.539657] Lustre: Mounted lustre-client
[ 5917.370749] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5924.407274] Lustre: lustre-MDT0000-mdc-ffff91a807e38800: Connection to lustre-MDT0000 (at 192.168.201.103@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5924.448135] Lustre: Skipped 8 previous similar messages
[ 5940.575147] Lustre: 2399:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773614791/real 1773614791]  req@ffff91a810375500 x1859763795275904/t0(0) o400->MGC192.168.201.103@tcp@192.168.201.103@tcp:26/25 lens 224/224 e 0 to 1 dl 1773614807 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5940.614622] Lustre: 2399:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5940.628827] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 192.168.201.103@tcp) was lost; in progress operations using this service will fail
[ 5950.956524] Lustre: Evicted from MGS (at 192.168.201.103@tcp) after server handle changed from 0xf6516185796836c to 0xf651618579685cd
[ 5950.971803] Lustre: MGC192.168.201.103@tcp: Connection restored to 192.168.201.103@tcp (at 192.168.201.103@tcp)
[ 5950.982879] Lustre: Skipped 8 previous similar messages
[ 5978.444090] LustreError: 72521:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff91a806de8000: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 5978.465893] LustreError: 72521:0:(lov_obd.c:783:lov_cleanup()) Skipped 1 previous similar message
[ 5978.570880] Lustre: Unmounted lustre-client
[ 5990.498616] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 18:47:34 (1773614854)
[ 6025.792847] LustreError: 2398:0:(client.c:3418:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff91a810377800 x1859763795293184/t94489280516(94489280516) o101->lustre-MDT0000-mdc-ffff91a807e38800@192.168.201.103@tcp:12/10 lens 536/664 e 0 to 0 dl 1773614908 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 6044.037864] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6046.718956] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6058.503581] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 18:48:42 (1773614922)
[ 6066.710556] LustreError: 2398:0:(osc_request.c:1020:osc_init_grant()) lustre-OST0000-osc-ffff91a807e38800: granted 8437760 but already consumed 12632064
[ 6066.727341] LustreError: lustre-OST0000-osc-ffff91a807e38800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6066.758835] Lustre: 2400:0:(llite_lib.c:4158:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.103@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 6067.148735] LustreError: 74547:0:(ldlm_resource.c:1170:ldlm_resource_complain()) lustre-OST0000-osc-ffff91a807e38800: namespace resource [0x280000401:0x1742:0x0].0x0 (ffff91a804527500) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6081.283390] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 18:49:05 (1773614945)
[ 6117.314502] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 18:49:41 (1773614981)
[ 6148.435840] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 18:50:12 (1773615012)
[ 6179.655555] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 18:50:44 (1773615044)
[ 6250.399505] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 18:51:54 (1773615114)
[ 6324.365879] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 18:53:08 (1773615188)
[ 6356.775407] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 18:53:41 (1773615221)
[ 6427.295910] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 18:54:51 (1773615291)
[ 6459.582254] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 18:55:23 (1773615323)
[ 6491.160071] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 18:55:55 (1773615355)
[ 6524.397617] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 18:56:28 (1773615388)
[ 6524.777460] LustreError: 81285:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff91a807e38800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 6524.785326] LustreError: 81285:0:(lov_obd.c:783:lov_cleanup()) Skipped 1 previous similar message
[ 6524.913724] Lustre: Unmounted lustre-client
[ 6592.957490] Lustre: Mounted lustre-client
[ 6615.750257] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 18:58:00 (1773615480)
[ 6639.463764] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 18:58:24 (1773615504)
[ 6644.208450] Lustre: lustre-MDT0000-mdc-ffff91a8182b3000: Connection to lustre-MDT0000 (at 192.168.201.103@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6644.247534] Lustre: Skipped 8 previous similar messages
[ 6665.695590] Lustre: 2400:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773615516/real 1773615516]  req@ffff91a807a18700 x1859763795431808/t0(0) o400->MGC192.168.201.103@tcp@192.168.201.103@tcp:26/25 lens 224/224 e 0 to 1 dl 1773615532 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6665.730626] Lustre: 2400:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
[ 6665.748481] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 192.168.201.103@tcp) was lost; in progress operations using this service will fail
[ 6665.776439] LustreError: Skipped 1 previous similar message
[ 6675.952170] Lustre: Evicted from MGS (at 192.168.201.103@tcp) after server handle changed from 0xf6516185796afdb to 0xf6516185796b529
[ 6675.972127] Lustre: Skipped 1 previous similar message
[ 6675.991485] Lustre: MGC192.168.201.103@tcp: Connection restored to 192.168.201.103@tcp (at 192.168.201.103@tcp)
[ 6676.024072] Lustre: Skipped 9 previous similar messages
[ 6693.357591] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 18:59:17 (1773615557)
[ 6730.692786] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 18:59:54 (1773615594)
[ 6731.289602] Lustre: *** cfs_fail_loc=51b, val=3***
[ 6745.514070] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 19:00:10 (1773615610)
[ 6746.314935] Lustre: *** cfs_fail_loc=51b, val=4***
[ 6761.208861] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 19:00:25 (1773615625)
[ 6762.016913] Lustre: *** cfs_fail_loc=50f, val=3***
[ 6777.130896] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 19:00:40 (1773615640)
[ 6777.655434] Lustre: *** cfs_fail_loc=50f, val=4***
[ 6792.668206] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 19:00:56 (1773615656)
[ 6793.322586] Lustre: *** cfs_fail_loc=510, val=3***
[ 6807.920330] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 19:01:12 (1773615672)
[ 6808.481327] Lustre: *** cfs_fail_loc=51b, val=3***
[ 6879.260520] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 19:02:23 (1773615743)
[ 6879.778155] Lustre: *** cfs_fail_loc=51c, val=3***
[ 6949.027680] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 19:03:33 (1773615813)
[ 6949.529964] LustreError: 89410:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 6952.977555] LustreError: lustre-MDT0000-mdc-ffff91a8182b3000: operation ldlm_enqueue to node 192.168.201.103@tcp failed: rc = -107
[ 6952.990512] LustreError: Skipped 4 previous similar messages
[ 6953.021890] LustreError: lustre-MDT0000-mdc-ffff91a8182b3000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6953.089814] LustreError: 89424:0:(ldlm_resource.c:1170:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff91a8182b3000: namespace resource [0x20000afe2:0xc:0x0].0xc (ffff91a804feb400) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6953.615258] LustreError: 89410:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 6955.825771] LustreError: 89440:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 6959.545367] LustreError: lustre-MDT0000-mdc-ffff91a8182b3000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6959.566755] LustreError: 89454:0:(ldlm_resource.c:1170:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff91a8182b3000: namespace resource [0x20000afe2:0xc:0x0].0xc (ffff91a804fe6600) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6959.620123] LustreError: 89454:0:(ldlm_resource.c:1170:ldlm_resource_complain()) Skipped 1 previous similar message
[ 6959.640018] LustreError: 89459:0:(file.c:6089:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 6959.660409] LustreError: 89459:0:(file.c:6089:ll_inode_revalidate_fini()) Skipped 9 previous similar messages
[ 6959.832017] LustreError: 89440:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 6967.398768] LustreError: lustre-MDT0000-mdc-ffff91a8182b3000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6968.114824] LustreError: 89496:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 6971.761074] LustreError: lustre-MDT0000-mdc-ffff91a8182b3000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6971.882130] LustreError: 89516:0:(file.c:6089:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 6972.119103] LustreError: 89496:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 6972.127470] LustreError: 89496:0:(file.c:249:ll_close_inode_openhandle()) lustre-clilmv-ffff91a8182b3000: inode [0x20000afe2:0xc:0x0] mdc close failed: rc = -108
[ 6974.313507] LustreError: 89533:0:(file.c:6089:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 6974.340220] LustreError: 89533:0:(file.c:6089:ll_inode_revalidate_fini()) Skipped 12 previous similar messages
[ 6979.704886] LustreError: 89573:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 6983.776344] LustreError: 89573:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 6984.614209] LustreError: lustre-MDT0000-mdc-ffff91a8182b3000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6991.858648] LustreError: lustre-MDT0000-mdc-ffff91a8182b3000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6993.115937] LustreError: 89630:0:(ldlm_flock.c:856:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 6995.988491] LustreError: lustre-MDT0000-mdc-ffff91a8182b3000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6996.045857] LustreError: 89644:0:(ldlm_resource.c:1170:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff91a8182b3000: namespace resource [0x20000afe2:0xc:0x0].0xc (ffff91a80460a000) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6996.076431] LustreError: 89644:0:(ldlm_resource.c:1170:ldlm_resource_complain()) Skipped 1 previous similar message
[ 6997.155099] LustreError: 89630:0:(ldlm_flock.c:856:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 7001.061069] LustreError: lustre-MDT0000-mdc-ffff91a8182b3000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7001.231703] LustreError: 89677:0:(file.c:6089:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 7001.275114] LustreError: 89677:0:(file.c:6089:ll_inode_revalidate_fini()) Skipped 10 previous similar messages
[ 7001.440173] LustreError: 89658:0:(file.c:249:ll_close_inode_openhandle()) lustre-clilmv-ffff91a8182b3000: inode [0x20000afe2:0xc:0x0] mdc close failed: rc = -108
[ 7010.553102] LustreError: 89718:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 7010.581280] LustreError: 89718:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 7014.575195] LustreError: 89718:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 7014.589520] LustreError: 89718:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 7015.915821] LustreError: lustre-MDT0000-mdc-ffff91a8182b3000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7029.380178] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 19:04:53 (1773615893)
[ 7090.979738] LustreError: 81777:0:(ldlm_lockd.c:2883:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 7095.071207] LustreError: 81777:0:(ldlm_lockd.c:2883:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 7104.601374] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 19:06:09 (1773615969)
[ 7174.454801] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 19:07:18 (1773616038)
[ 7230.522243] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 19:08:15 (1773616095)
[ 7271.239922] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 19:08:55 (1773616135)
[ 7272.336346] Lustre: Mounted lustre-client
[ 7398.370516] LustreError: 93474:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff91a81094d000: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 7398.384381] LustreError: 93474:0:(lov_obd.c:783:lov_cleanup()) Skipped 1 previous similar message
[ 7398.498152] Lustre: Unmounted lustre-client
[ 7409.832506] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 19:11:13 (1773616273)
[ 7410.826120] LustreError: 94097:0:(osc_request.c:2983:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 4000ms
[ 7414.841128] LustreError: 94097:0:(osc_request.c:2983:osc_build_rpc()) cfs_fail_timeout id 414 awake
[ 7415.231902] LustreError: lustre-OST0000-osc-ffff91a8182b3000: operation ldlm_enqueue to node 192.168.201.103@tcp failed: rc = -107
[ 7415.253755] LustreError: Skipped 8 previous similar messages
[ 7415.256264] Lustre: lustre-OST0000-osc-ffff91a8182b3000: Connection to lustre-OST0000 (at 192.168.201.103@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7415.277229] Lustre: Skipped 14 previous similar messages
[ 7415.294858] LustreError: lustre-OST0000-osc-ffff91a8182b3000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7418.300349] Lustre: DEBUG MARKER: recovery-small test_131: @@@@@@ FAIL: dd succeeded