[ 2172.380547] LustreError: lustre-OST0000-osc-ffff9a0fa0486000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2172.396967] Lustre: 2348:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.109@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 2182.884313] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 17:33:01 (1785879181)
[ 2185.274841] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 2187.945244] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 17:33:07 (1785879187)
[ 2190.245963] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 2193.310933] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 17:33:12 (1785879192)
[ 2198.307147] LustreError: lustre-MDT0000-mdc-ffff9a0fa0486000: operation mds_reint to node 192.168.204.109@tcp failed: rc = -19
[ 2198.334965] LustreError: Skipped 1 previous similar message
[ 2215.904912] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 192.168.204.109@tcp) was lost; in progress operations using this service will fail
[ 2226.155663] Lustre: Evicted from MGS (at 192.168.204.109@tcp) after server handle changed from 0x7bb61e041d520331 to 0x7bb61e041d521823
[ 2227.759187] Lustre: 15974:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.204.109@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2339.664106] LustreError: lustre-MDT0000-mdc-ffff9a0fa0486000: operation ldlm_enqueue to node 192.168.204.109@tcp failed: rc = -19
[ 2359.263660] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 192.168.204.109@tcp) was lost; in progress operations using this service will fail
[ 2369.461786] Lustre: Evicted from MGS (at 192.168.204.109@tcp) after server handle changed from 0x7bb61e041d521823 to 0x7bb61e041d53aa1e
[ 2369.477362] Lustre: MGC192.168.204.109@tcp: Connection restored to 192.168.204.109@tcp (at 192.168.204.109@tcp)
[ 2369.488229] Lustre: Skipped 12 previous similar messages
[ 2387.863995] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 17:36:27 (1785879387)
[ 2388.822447] Lustre: *** cfs_fail_loc=305, val=0***
[ 2388.824643] Lustre: Skipped 2 previous similar messages
[ 2428.895223] Lustre: 2348:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785879414/real 1785879414]  req@ffff9a0fa0a17800 x1872627810141056/t0(0) o400->MGC192.168.204.109@tcp@192.168.204.109@tcp:26/25 lens 224/224 e 0 to 1 dl 1785879430 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2428.935315] Lustre: 2348:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 12 previous similar messages
[ 2428.948119] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 192.168.204.109@tcp) was lost; in progress operations using this service will fail
[ 2439.181703] Lustre: Evicted from MGS (at 192.168.204.109@tcp) after server handle changed from 0x7bb61e041d53aa1e to 0x7bb61e041d53af9d
[ 2441.055614] Lustre: lustre-MDT0000-mdc-ffff9a0fa0486000: Connection to lustre-MDT0000 (at 192.168.204.109@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2441.088454] Lustre: Skipped 12 previous similar messages
[ 2461.029351] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2462.974284] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2472.175790] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 17:37:51 (1785879471)
[ 2490.343386] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 192.168.204.109@tcp) was lost; in progress operations using this service will fail
[ 2490.380383] Lustre: Evicted from MGS (at 192.168.204.109@tcp) after server handle changed from 0x7bb61e041d53af9d to 0x7bb61e041d53b569
[ 2490.450076] LustreError: lustre-MDT0000-mdc-ffff9a0fa0486000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2520.593544] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 17:38:39 (1785879519)
[ 2537.020678] LustreError: lustre-OST0000-osc-ffff9a0fa0486000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2566.859791] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 17:39:24 (1785879564)
[ 2582.290660] LustreError: lustre-MDT0000-mdc-ffff9a0fa0486000: operation mds_reint to node 192.168.204.109@tcp failed: rc = -19
[ 2602.975334] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 192.168.204.109@tcp) was lost; in progress operations using this service will fail
[ 2613.196171] Lustre: Evicted from MGS (at 192.168.204.109@tcp) after server handle changed from 0x7bb61e041d53b569 to 0x7bb61e041d53e4e8
[ 2634.367694] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2637.409264] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2724.831650] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 192.168.204.109@tcp) was lost; in progress operations using this service will fail
[ 2735.023255] Lustre: Evicted from MGS (at 192.168.204.109@tcp) after server handle changed from 0x7bb61e041d53e4e8 to 0x7bb61e041d555078
[ 2757.491531] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2759.656794] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2826.994592] LustreError: lustre-MDT0000-mdc-ffff9a0fa0486000: operation mds_reint to node 192.168.204.109@tcp failed: rc = -19
[ 2827.016017] LustreError: Skipped 1 previous similar message
[ 2846.687294] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 192.168.204.109@tcp) was lost; in progress operations using this service will fail
[ 2856.877216] Lustre: Evicted from MGS (at 192.168.204.109@tcp) after server handle changed from 0x7bb61e041d555078 to 0x7bb61e041d56ac79
[ 2879.215967] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2881.974646] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2921.087432] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 17:45:19 (1785879919)
[ 2949.087472] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 192.168.204.109@tcp) was lost; in progress operations using this service will fail
[ 2959.272490] Lustre: Evicted from MGS (at 192.168.204.109@tcp) after server handle changed from 0x7bb61e041d56ac79 to 0x7bb61e041d575a1b
[ 2969.756655] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 3005.370700] Lustre: MGC192.168.204.109@tcp: Connection restored to 192.168.204.109@tcp (at 192.168.204.109@tcp)
[ 3005.385388] Lustre: Skipped 14 previous similar messages
[ 3016.981040] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 3046.111567] Lustre: 2346:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785880031/real 1785880031]  req@ffff9a0f916a7100 x1872627813554944/t0(0) o400->MGC192.168.204.109@tcp@192.168.204.109@tcp:26/25 lens 224/224 e 0 to 1 dl 1785880047 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3046.154548] Lustre: 2346:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 10 previous similar messages
[ 3071.002635] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 3085.766597] LustreError: lustre-MDT0000-mdc-ffff9a0fa0486000: operation mds_reint to node 192.168.204.109@tcp failed: rc = -19
[ 3085.769631] Lustre: lustre-MDT0000-mdc-ffff9a0fa0486000: Connection to lustre-MDT0000 (at 192.168.204.109@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3085.775231] LustreError: Skipped 7 previous similar messages
[ 3085.806987] Lustre: Skipped 8 previous similar messages
[ 3101.664582] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 192.168.204.109@tcp) was lost; in progress operations using this service will fail
[ 3101.685384] LustreError: Skipped 2 previous similar messages
[ 3112.057562] Lustre: Evicted from MGS (at 192.168.204.109@tcp) after server handle changed from 0x7bb61e041d57a879 to 0x7bb61e041d580460
[ 3112.067605] Lustre: Skipped 2 previous similar messages
[ 3113.395677] Lustre: 15974:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.204.109@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 3126.633950] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 3179.188263] Lustre: 15974:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.204.109@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 3193.001825] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 3255.659189] Lustre: 15974:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.204.109@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 3269.022197] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 3372.645602] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 17:52:52 (1785880372)
[ 3438.831430] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3442.563922] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3720.096298] LustreError: lustre-OST0000-osc-ffff9a0fa0486000: operation ldlm_enqueue to node 192.168.204.109@tcp failed: rc = -107
[ 3720.105475] LustreError: Skipped 4 previous similar messages
[ 3720.112108] Lustre: lustre-OST0000-osc-ffff9a0fa0486000: Connection to lustre-OST0000 (at 192.168.204.109@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3720.128994] Lustre: Skipped 4 previous similar messages
[ 3750.909599] Lustre: lustre-OST0000-osc-ffff9a0fa0486000: Connection restored to 192.168.204.109@tcp (at 192.168.204.109@tcp)
[ 3750.931186] Lustre: Skipped 12 previous similar messages
[ 3785.105297] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3787.996695] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4099.528392] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4101.923359] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4347.370277] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 18:09:06 (1785881346)
[ 4364.767792] Lustre: 50636:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785881350/real 1785881350]  req@ffff9a0f91403100 x1872627828479488/t0(0) o101->lustre-MDT0000-mdc-ffff9a0fa0486000@192.168.204.109@tcp:12/10 lens 576/1152 e 0 to 1 dl 1785881366 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 4364.823496] Lustre: 50636:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
[ 4364.840552] Lustre: lustre-MDT0000-mdc-ffff9a0fa0486000: Connection to lustre-MDT0000 (at 192.168.204.109@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4364.873695] Lustre: Skipped 1 previous similar message
[ 4364.950659] Lustre: lustre-MDT0000-mdc-ffff9a0fa0486000: Connection restored to 192.168.204.109@tcp (at 192.168.204.109@tcp)
[ 4364.969024] Lustre: Skipped 1 previous similar message
[ 4374.653527] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 18:09:34 (1785881374)
[ 4404.328359] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 18:10:03 (1785881403)
[ 4435.523822] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 18:10:35 (1785881435)
[ 4436.624833] Lustre: Mounted lustre-client - version 2.17.56_50_g55fd350
[ 4467.679139] Lustre: 2348:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785881453/real 1785881453]  req@ffff9a0f9138ea00 x1872627828497536/t0(0) o400->lustre-MDT0000-mdc-ffff9a0fa0486000@192.168.204.109@tcp:12/10 lens 224/224 e 0 to 1 dl 1785881469 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4467.727132] Lustre: 2348:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 4467.752827] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 192.168.204.109@tcp) was lost; in progress operations using this service will fail
[ 4467.766242] LustreError: Skipped 3 previous similar messages
[ 4477.947854] Lustre: Evicted from MGS (at 192.168.204.109@tcp) after server handle changed from 0x7bb61e041d5994a9 to 0x7bb61e041d692347
[ 4477.977666] Lustre: Skipped 3 previous similar messages
[ 4501.985166] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4504.081351] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4506.546450] Lustre: Unmounted lustre-client
[ 4516.323496] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 18:11:54 (1785881514)
[ 4626.400718] Lustre: 2349:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785881611/real 1785881611]  req@ffff9a0f9138f100 x1872627828533376/t0(0) o4->lustre-OST0000-osc-ffff9a0fa0486000@192.168.204.109@tcp:6/4 lens 488/448 e 0 to 1 dl 1785881627 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4626.428809] Lustre: 2349:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 50 previous similar messages
[ 4826.960140] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 18:17:06 (1785881826)
[ 4882.135533] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 18:18:01 (1785881881)
[ 4886.953917] Lustre: Unmounted lustre-client
[ 4923.001937] Lustre: Mounted lustre-client - version 2.17.56_50_g55fd350
[ 4935.827631] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 18:18:54 (1785881934)
[ 4936.365513] LustreError: 56574:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4937.457598] LustreError: 56574:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4937.817514] Lustre: *** cfs_fail_loc=305, val=0***
[ 4963.116683] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 18:19:22 (1785881962)
[ 4963.781069] Lustre: Mounted lustre-client - version 2.17.56_50_g55fd350
[ 4968.488374] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4978.853598] Lustre: Unmounted lustre-client
[ 4988.113636] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 18:19:47 (1785881987)
[ 5084.494012] LustreError: lustre-MDT0000-mdc-ffff9a0f914f4800: operation mds_reint to node 192.168.204.109@tcp failed: rc = -19
[ 5084.502840] LustreError: Skipped 1 previous similar message
[ 5084.510587] Lustre: lustre-MDT0000-mdc-ffff9a0f914f4800: Connection to lustre-MDT0000 (at 192.168.204.109@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5084.529168] Lustre: Skipped 23 previous similar messages
[ 5104.095185] Lustre: 2347:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785882089/real 1785882089]  req@ffff9a0f887bea00 x1872627831193088/t0(0) o400->MGC192.168.204.109@tcp@192.168.204.109@tcp:26/25 lens 224/224 e 0 to 1 dl 1785882105 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5104.150707] Lustre: 2347:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 86 previous similar messages
[ 5104.169827] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 192.168.204.109@tcp) was lost; in progress operations using this service will fail
[ 5114.405025] Lustre: Evicted from MGS (at 192.168.204.109@tcp) after server handle changed from 0x7bb61e041d693fef to 0x7bb61e041d6beeba
[ 5114.430992] Lustre: MGC192.168.204.109@tcp: Connection restored to 192.168.204.109@tcp (at 192.168.204.109@tcp)
[ 5114.438129] Lustre: Skipped 24 previous similar messages
[ 5331.514294] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 18:25:30 (1785882330)
[ 5337.729344] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5358.568842] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 192.168.204.109@tcp) was lost; in progress operations using this service will fail
[ 5358.601342] Lustre: Evicted from MGS (at 192.168.204.109@tcp) after server handle changed from 0x7bb61e041d6beeba to 0x7bb61e041d70c3c2
[ 5358.741990] LustreError: lustre-MDT0000-mdc-ffff9a0f914f4800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5380.045690] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 18:26:18 (1785882378)
[ 5380.910409] Lustre: Mounted lustre-client - version 2.17.56_50_g55fd350
[ 5389.105316] LustreError: lustre-OST0000-osc-ffff9a0f914f4800: operation ldlm_enqueue to node 192.168.204.109@tcp failed: rc = -107
[ 5389.158519] LustreError: lustre-OST0000-osc-ffff9a0f914f4800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5404.519933] Lustre: Unmounted lustre-client
[ 5413.742865] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 18:26:53 (1785882413)
[ 5423.322091] LustreError: lustre-MDT0000-mdc-ffff9a0f914f4800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5423.357179] LustreError: 61001:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 5437.384665] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 18:27:16 (1785882436)
[ 5437.740957] LustreError: 61685:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 5442.947246] LustreError: lustre-MDT0000-mdc-ffff9a0f914f4800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5443.006797] LustreError: 61707:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 5443.046268] LustreError: 61685:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=1
[ 5443.086540] LustreError: 61685:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting
[ 5443.123284] LustreError: 61712:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5444.275802] LustreError: 61718:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5444.308027] LustreError: 61718:0:(file.c:6154:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5446.885253] LustreError: 61741:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5446.900405] LustreError: 61741:0:(file.c:6154:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5451.094128] LustreError: 61774:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5451.126280] LustreError: 61774:0:(file.c:6154:ll_inode_revalidate_fini()) Skipped 20 previous similar messages
[ 5464.871186] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 18:27:44 (1785882464)
[ 5516.733624] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5518.900279] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5532.909962] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 18:28:52 (1785882532)
[ 5585.331569] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5587.851538] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5601.600690] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 18:30:01 (1785882601)
[ 5672.215467] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5674.296769] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5686.705513] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 18:31:25 (1785882685)
[ 5695.470589] Lustre: lustre-OST0000-osc-ffff9a0f914f4800: Connection to lustre-OST0000 (at 192.168.204.109@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5695.491638] Lustre: Skipped 9 previous similar messages
[ 5716.011440] Lustre: lustre-OST0000-osc-ffff9a0f914f4800: Connection restored to 192.168.204.109@tcp (at 192.168.204.109@tcp)
[ 5716.022445] Lustre: Skipped 10 previous similar messages
[ 5738.215216] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5741.441882] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5753.972757] Lustre: Unmounted lustre-client
[ 5828.813247] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5831.861977] Lustre: Mounted lustre-client - version 2.17.56_50_g55fd350
[ 5839.884992] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 18:33:59 (1785882839)
[ 5843.604447] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5845.905748] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 18:34:05 (1785882845)
[ 5883.889578] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 18:34:43 (1785882883)
[ 5885.496435] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5887.073141] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 18:34:47 (1785882887)
[ 5888.423300] Lustre: *** cfs_fail_loc=805, val=0***
[ 5888.479201] Lustre: Mounted lustre-client - version 2.17.56_50_g55fd350
[ 5894.422354] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5915.103154] Lustre: 2347:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785882900/real 1785882900]  req@ffff9a0fa0c46300 x1872627833269120/t0(0) o400->lustre-MDT0000-mdc-ffff9a0f8a091800@192.168.204.109@tcp:12/10 lens 224/224 e 0 to 1 dl 1785882916 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5915.152515] Lustre: 2347:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5915.164606] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 192.168.204.109@tcp) was lost; in progress operations using this service will fail
[ 5925.358437] Lustre: Evicted from MGS (at 192.168.204.109@tcp) after server handle changed from 0x7bb61e041d70cde7 to 0x7bb61e041d70d0f7
[ 5936.824023] Lustre: Unmounted lustre-client
[ 5948.074369] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 18:35:48 (1785882948)
[ 5990.399600] LustreError: 2345:0:(client.c:3435:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff9a0f884a7480 x1872627833281408/t94489280516(94489280516) o101->lustre-MDT0000-mdc-ffff9a0f8a091800@192.168.204.109@tcp:12/10 lens 664/608 e 0 to 0 dl 1785883007 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 6001.742673] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6004.210281] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6017.186414] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 18:36:56 (1785883016)
[ 6018.821659] LustreError: lustre-OST0000-osc-ffff9a0f8a091800: operation ost_write to node 192.168.204.109@tcp failed: rc = -107
[ 6018.827896] LustreError: Skipped 2 previous similar messages
[ 6018.862737] LustreError: lustre-OST0000-osc-ffff9a0f8a091800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6018.886766] Lustre: 2346:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.109@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 6018.933346] LustreError: 72898:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff9a0f8a091800: namespace resource [0x240000400:0x3382:0x0].0x0 (ffff9a0f98094b00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6030.757753] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 18:37:10 (1785883030)
[ 6032.762200] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 6035.541294] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 18:37:14 (1785883034)
[ 6037.366220] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 6040.330803] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 18:37:19 (1785883039)
[ 6042.660191] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 6045.403717] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 18:37:24 (1785883044)
[ 6048.075923] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 6052.388982] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 18:37:30 (1785883050)
[ 6054.742480] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 6057.220677] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 18:37:36 (1785883056)
[ 6059.726666] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 6061.511689] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 18:37:41 (1785883061)
[ 6064.737415] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 6067.520830] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 18:37:46 (1785883066)
[ 6069.670872] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 6074.401496] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 18:37:52 (1785883072)
[ 6076.793425] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 6080.526639] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 18:37:58 (1785883078)
[ 6083.969681] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 6087.356828] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 18:38:05 (1785883085)
[ 6089.144669] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 6090.832936] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 18:38:10 (1785883090)
[ 6093.555405] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 6095.027253] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 18:38:15 (1785883095)
[ 6140.622179] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 18:39:00 (1785883140)
[ 6172.160744] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 18:39:31 (1785883171)
[ 6173.101806] Lustre: *** cfs_fail_loc=51b, val=3***
[ 6187.879646] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 18:39:47 (1785883187)
[ 6188.854853] Lustre: *** cfs_fail_loc=51b, val=4***
[ 6206.439697] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 18:40:05 (1785883205)
[ 6206.885779] Lustre: *** cfs_fail_loc=50f, val=3***
[ 6219.214966] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 18:40:18 (1785883218)
[ 6219.529951] Lustre: *** cfs_fail_loc=50f, val=4***
[ 6230.303351] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 18:40:29 (1785883229)
[ 6230.758302] Lustre: *** cfs_fail_loc=510, val=3***
[ 6242.568894] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 18:40:41 (1785883241)
[ 6243.211385] Lustre: *** cfs_fail_loc=51b, val=3***
[ 6301.663227] Lustre: lustre-OST0001-osc-ffff9a0f8a091800: Connection to lustre-OST0001 (at 192.168.204.109@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6301.702668] Lustre: Skipped 5 previous similar messages
[ 6312.168633] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 18:41:51 (1785883311)
[ 6312.732240] Lustre: *** cfs_fail_loc=51c, val=3***
[ 6384.825736] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 18:43:03 (1785883383)
[ 6385.455280] LustreError: 83129:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 6389.139394] LustreError: lustre-MDT0000-mdc-ffff9a0f8a091800: operation ldlm_enqueue to node 192.168.204.109@tcp failed: rc = -107
[ 6389.166233] LustreError: lustre-MDT0000-mdc-ffff9a0f8a091800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6389.219650] LustreError: 83144:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9a0f8a091800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9a0f980fc900) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6389.283156] Lustre: lustre-MDT0000-mdc-ffff9a0f8a091800: Connection restored to 192.168.204.109@tcp (at 192.168.204.109@tcp)
[ 6389.307970] Lustre: Skipped 7 previous similar messages
[ 6389.479719] LustreError: 83129:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 6391.742755] LustreError: 83159:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 6394.962338] LustreError: lustre-MDT0000-mdc-ffff9a0f8a091800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6394.991510] LustreError: 83174:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9a0f8a091800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9a0f980fc900) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6395.021539] LustreError: 83174:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 6395.839829] LustreError: 83159:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 6403.311247] LustreError: lustre-MDT0000-mdc-ffff9a0f8a091800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6403.771550] LustreError: 83215:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 6407.297931] LustreError: lustre-MDT0000-mdc-ffff9a0f8a091800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6407.420154] LustreError: 83235:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 6407.432244] LustreError: 83235:0:(file.c:6154:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 6407.792048] LustreError: 83215:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 6407.801294] LustreError: 83215:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9a0f8a091800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 6408.549383] LustreError: 83241:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 6408.573596] LustreError: 83241:0:(file.c:6154:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 6411.127607] LustreError: 83264:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 6411.153589] LustreError: 83264:0:(file.c:6154:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 6411.362762] LustreError: 83230:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9a0f8a091800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff9a0f98094200) refcount nonzero (2) after lock cleanup; forcing cleanup.
[ 6411.383033] LustreError: 83230:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 6415.060909] LustreError: 83290:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 6419.087110] LustreError: 83290:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 6419.402947] LustreError: lustre-MDT0000-mdc-ffff9a0f8a091800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6427.139797] LustreError: lustre-MDT0000-mdc-ffff9a0f8a091800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6428.030212] LustreError: 83347:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 6431.345735] LustreError: lustre-MDT0000-mdc-ffff9a0f8a091800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6431.374767] LustreError: 83362:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9a0f8a091800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9a0f88773f00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6431.426407] LustreError: 83362:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 6432.071173] LustreError: 83347:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 6434.938745] LustreError: lustre-MDT0000-mdc-ffff9a0f8a091800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6435.014524] LustreError: 83394:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 6435.031604] LustreError: 83394:0:(file.c:6154:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 6436.250965] LustreError: 83375:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9a0f8a091800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 6444.563739] LustreError: 83446:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 6444.577591] LustreError: 83446:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 6448.513756] LustreError: lustre-MDT0000-mdc-ffff9a0f8a091800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6448.568772] LustreError: 83446:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 6448.585300] LustreError: 83446:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 6448.605249] LustreError: 83469:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 6448.611694] LustreError: 83446:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9a0f8a091800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 6448.624184] LustreError: 83469:0:(file.c:6154:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 6462.148595] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 18:44:21 (1785883461)
[ 6482.810195] LustreError: 68351:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 6482.816967] LustreError: 68351:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 6486.879094] LustreError: 68351:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 6486.898402] LustreError: 68351:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 6496.255098] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 18:44:55 (1785883495)
[ 6539.745651] Lustre: 84898:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785883500/real 1785883500]  req@ffff9a0f911cfb80 x1872627833409920/t0(0) o101->lustre-MDT0000-mdc-ffff9a0f8a091800@192.168.204.109@tcp:12/10 lens 576/1152 e 1 to 1 dl 1785883541 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'stat.0' uid:0 gid:0 projid:0
[ 6539.794832] Lustre: 84898:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 20 previous similar messages
[ 6549.442456] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 18:45:48 (1785883548)
[ 6615.887781] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 18:46:55 (1785883615)
[ 6657.436769] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 18:47:35 (1785883655)
[ 6658.315650] Lustre: Mounted lustre-client - version 2.17.56_50_g55fd350
[ 6782.992950] Lustre: Unmounted lustre-client
[ 6790.553327] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 18:49:50 (1785883790)
[ 6791.065952] LustreError: 87776:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 6794.349910] LustreError: lustre-OST0000-osc-ffff9a0f8a091800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6794.392703] LustreError: 87864:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff9a0f8a091800: namespace resource [0x240000400:0x33ad:0x0].0x0 (ffff9a0f980fc600) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6794.425105] LustreError: 87864:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 6794.751172] LustreError: 87776:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout interrupted
[ 6794.759586] Lustre: 2347:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.109@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 6805.060025] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 18:50:04 (1785883804)
[ 6873.912158] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 18:51:12 (1785883872)
[ 6876.750805] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 6880.174909] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 18:51:18 (1785883878)
[ 6937.577988] Lustre: lustre-MDT0000-mdc-ffff9a0f8a091800: Connection to lustre-MDT0000 (at 192.168.204.109@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6937.615853] Lustre: Skipped 15 previous similar messages
[ 6947.102693] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6949.244085] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 18:52:28 (1785883948)
[ 7006.193884] Lustre: lustre-MDT0000-mdc-ffff9a0f8a091800: Connection restored to 192.168.204.109@tcp (at 192.168.204.109@tcp)
[ 7006.201697] Lustre: Skipped 14 previous similar messages
[ 7009.148266] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 18:53:27 (1785884007)
[ 7012.936566] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 7015.485811] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 18:53:35 (1785884015)
[ 7019.059962] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 7021.205818] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 18:53:41 (1785884021)
[ 7066.078503] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 18:54:25 (1785884065)
[ 7082.006121] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7113.697278] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 192.168.204.109@tcp) was lost; in progress operations using this service will fail
[ 7113.719809] LustreError: Skipped 2 previous similar messages
[ 7123.963272] Lustre: Evicted from MGS (at 192.168.204.109@tcp) after server handle changed from 0x7bb61e041d70dc50 to 0x7bb61e041d70f475
[ 7123.972290] Lustre: Skipped 2 previous similar messages
[ 7125.482332] Lustre: 68354:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.204.109@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 7142.372034] Lustre: 2349:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785884120/real 1785884120]  req@ffff9a0f887bca80 x1872627833501056/t0(0) o400->lustre-MDT0000-mdc-ffff9a0f8a091800@192.168.204.109@tcp:12/10 lens 224/224 e 0 to 1 dl 1785884143 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7142.431628] Lustre: 2349:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 9 previous similar messages
[ 7150.718444] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 7152.350666] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 7168.560656] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 18:56:08 (1785884168)
[ 7172.557191] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 7175.038971] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 18:56:14 (1785884174)
[ 7192.550422] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 192.168.204.109@tcp) was lost; in progress operations using this service will fail
[ 7192.584234] Lustre: Evicted from MGS (at 192.168.204.109@tcp) after server handle changed from 0x7bb61e041d70f475 to 0x7bb61e041d70f930
[ 7212.052393] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 18:56:51 (1785884211)
[ 7259.120718] Lustre: Evicted from MGS (at 192.168.204.109@tcp) after server handle changed from 0x7bb61e041d70f930 to 0x7bb61e041d70fcb7
[ 7283.115862] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 18:58:02 (1785884282)
[ 7293.411507] LustreError: lustre-OST0000-osc-ffff9a0f8a091800: operation ost_setattr to node 192.168.204.109@tcp failed: rc = -19
[ 7293.432295] LustreError: Skipped 9 previous similar messages
[ 7365.474754] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 7368.464228] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 7453.663616] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 192.168.204.109@tcp) was lost; in progress operations using this service will fail
[ 7453.673686] LustreError: Skipped 1 previous similar message
[ 7464.118029] Lustre: Evicted from MGS (at 192.168.204.109@tcp) after server handle changed from 0x7bb61e041d70fcb7 to 0x7bb61e041d710c70
[ 7477.487679] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 7479.907060] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 7531.674463] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 7533.121561] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 7710.741746] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 19:05:10 (1785884710)
[ 7727.510821] Lustre: lustre-OST0000-osc-ffff9a0f8a091800: Connection to lustre-OST0000 (at 192.168.204.109@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7727.536357] Lustre: Skipped 7 previous similar messages
[ 7798.491400] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 7801.749493] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 8118.119879] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 19:11:57 (1785885117)
[ 8289.780717] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 192.168.204.109@tcp) was lost; in progress operations using this service will fail
[ 8289.791734] LustreError: Skipped 1 previous similar message
[ 8289.800414] Lustre: Evicted from MGS (at 192.168.204.109@tcp) after server handle changed from 0x7bb61e041d710e30 to 0x7bb61e041d80b0b6
[ 8289.812706] Lustre: Skipped 1 previous similar message
[ 8289.820331] Lustre: MGC192.168.204.109@tcp: Connection restored to 192.168.204.109@tcp (at 192.168.204.109@tcp)
[ 8289.828714] Lustre: Skipped 10 previous similar messages
[ 8329.783326] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 19:15:29 (1785885329)
[ 8331.441836] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 8334.484274] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 19:15:33 (1785885333)
[ 8337.700315] LustreError: lustre-MDT0000-mdc-ffff9a0f8a091800: operation ldlm_enqueue to node 192.168.204.109@tcp failed: rc = -107
[ 8337.717016] LustreError: Skipped 359 previous similar messages
[ 8337.735135] Lustre: lustre-MDT0000-mdc-ffff9a0f8a091800: Connection to lustre-MDT0000 (at 192.168.204.109@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 8337.769840] Lustre: Skipped 1 previous similar message
[ 8337.794919] LustreError: lustre-MDT0000-mdc-ffff9a0f8a091800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 8337.832897] LustreError: 102175:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9a0f8a091800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff9a0f91a85000) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 8337.855766] LustreError: 102175:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 8346.971558] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 19:15:46 (1785885346)
[ 8515.055212] LustreError: lustre-OST0000-osc-ffff9a0f8a091800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 8517.569982] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 19:18:36 (1785885516)
[ 8545.759216] Lustre: 2348:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785885526/real 1785885526]  req@ffff9a0fac142300 x1872627851224704/t0(0) o4->lustre-OST0000-osc-ffff9a0f8a091800@192.168.204.109@tcp:6/4 lens 4584/448 e 0 to 1 dl 1785885546 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 8545.793891] Lustre: 2348:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 21 previous similar messages
[ 8545.827911] Lustre: lustre-OST0000-osc-ffff9a0f8a091800: Connection restored to 192.168.204.109@tcp (at 192.168.204.109@tcp)
[ 8545.834950] Lustre: Skipped 2 previous similar messages
[ 8565.924302] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 19:19:25 (1785885565)
[ 8568.072830] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 8571.601918] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 19:19:30 (1785885570)
[ 8573.954858] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 8577.121911] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 19:19:35 (1785885575)
[ 8623.701923] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 19:20:21 (1785885621)
[ 8673.759256] Lustre: 2346:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785885659/real 1785885659]  req@ffff9a0fb3011f80 x1872627851381504/t0(0) o400->MGC192.168.204.109@tcp@192.168.204.109@tcp:26/25 lens 224/224 e 0 to 1 dl 1785885675 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 8673.794536] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 192.168.204.109@tcp) was lost; in progress operations using this service will fail
[ 8695.372730] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 19:21:34 (1785885694)
[ 8697.546512] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 8699.356557] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 19:21:39 (1785885699)
[ 8702.286028] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 8705.013152] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 19:21:44 (1785885704)
[ 8725.318675] Lustre: Unmounted lustre-client
[ 8727.706399] Lustre: Mounted lustre-client - version 2.17.56_50_g55fd350
[ 8730.871493] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 8753.695236] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 192.168.204.109@tcp) was lost; in progress operations using this service will fail
[ 8753.722712] Lustre: MGC192.168.204.109@tcp: Connection restored to 192.168.204.109@tcp (at 192.168.204.109@tcp)
[ 8753.729869] Lustre: Skipped 2 previous similar messages
[ 8755.370106] Lustre: 106959:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.204.109@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 8777.994248] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2