[ 2155.851843] LustreError: lustre-OST0000-osc-ffff96fd057df000: operation ost_sync to node 192.168.201.103@tcp failed: rc = -107
[ 2155.922157] LustreError: lustre-OST0000-osc-ffff96fd057df000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2155.994965] Lustre: 2355:0:(llite_lib.c:4198:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.103@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 2173.815675] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 17:51:07 (1781819467)
[ 2179.261801] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 2184.408976] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 17:51:17 (1781819477)
[ 2188.991963] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 2193.587116] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 17:51:27 (1781819487)
[ 2200.743431] LustreError: lustre-MDT0000-mdc-ffff96fd057df000: operation ldlm_enqueue to node 192.168.201.103@tcp failed: rc = -19
[ 2200.791667] Lustre: lustre-MDT0000-mdc-ffff96fd057df000: Connection to lustre-MDT0000 (at 192.168.201.103@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2200.805045] LustreError: Skipped 2 previous similar messages
[ 2200.870794] Lustre: Skipped 10 previous similar messages
[ 2219.487622] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 192.168.201.103@tcp) was lost; in progress operations using this service will fail
[ 2244.080314] Lustre: Evicted from MGS (at 192.168.201.103@tcp) after server handle changed from 0xaa643b29f949c2e6 to 0xaa643b29f949d9b4
[ 2244.156107] Lustre: MGC192.168.201.103@tcp: Connection restored to 192.168.201.103@tcp (at 192.168.201.103@tcp)
[ 2244.210794] Lustre: Skipped 9 previous similar messages
[ 2362.458163] LustreError: lustre-MDT0000-mdc-ffff96fd057df000: operation ldlm_enqueue to node 192.168.201.103@tcp failed: rc = -19
[ 2382.303947] Lustre: 2357:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781819663/real 1781819663]  req@ffff96fd18a09180 x1868370982087296/t0(0) o400->MGC192.168.201.103@tcp@192.168.201.103@tcp:26/25 lens 224/224 e 0 to 1 dl 1781819679 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2382.386551] Lustre: 2357:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 10 previous similar messages
[ 2382.409651] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 192.168.201.103@tcp) was lost; in progress operations using this service will fail
[ 2391.634314] Lustre: Evicted from MGS (at 192.168.201.103@tcp) after server handle changed from 0xaa643b29f949d9b4 to 0xaa643b29f94abd57
[ 2419.276815] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 17:55:14 (1781819714)
[ 2420.304588] Lustre: *** cfs_fail_loc=305, val=0***
[ 2420.309573] Lustre: Skipped 2 previous similar messages
[ 2441.510555] LustreError: lustre-OST0000-osc-ffff96fd057df000: operation ost_connect to node 192.168.201.103@tcp failed: rc = -75
[ 2468.326208] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 192.168.201.103@tcp) was lost; in progress operations using this service will fail
[ 2478.579308] Lustre: Evicted from MGS (at 192.168.201.103@tcp) after server handle changed from 0xaa643b29f94abd57 to 0xaa643b29f94ad480
[ 2479.763258] Lustre: 15888:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.201.103@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2504.299667] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2507.057591] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2522.861705] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 17:56:57 (1781819817)
[ 2545.148259] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 192.168.201.103@tcp) was lost; in progress operations using this service will fail
[ 2545.187178] Lustre: Evicted from MGS (at 192.168.201.103@tcp) after server handle changed from 0xaa643b29f94ad480 to 0xaa643b29f94ada99
[ 2545.307564] LustreError: lustre-MDT0000-mdc-ffff96fd057df000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2582.219032] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 17:57:57 (1781819877)
[ 2629.444415] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 17:58:44 (1781819924)
[ 2644.225339] LustreError: lustre-MDT0000-mdc-ffff96fd057df000: operation ldlm_enqueue to node 192.168.201.103@tcp failed: rc = -19
[ 2662.881752] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 192.168.201.103@tcp) was lost; in progress operations using this service will fail
[ 2673.134855] Lustre: Evicted from MGS (at 192.168.201.103@tcp) after server handle changed from 0xaa643b29f94ada99 to 0xaa643b29f94b154e
[ 2696.078386] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2699.716104] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2768.612323] LustreError: lustre-MDT0000-mdc-ffff96fd057df000: operation mds_reint to node 192.168.201.103@tcp failed: rc = -19
[ 2784.735903] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 192.168.201.103@tcp) was lost; in progress operations using this service will fail
[ 2795.003989] Lustre: Evicted from MGS (at 192.168.201.103@tcp) after server handle changed from 0xaa643b29f94b154e to 0xaa643b29f94c1d2c
[ 2796.289123] Lustre: 15888:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.201.103@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2824.877720] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2828.268469] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2895.813303] LustreError: lustre-MDT0000-mdc-ffff96fd057df000: operation mds_reint to node 192.168.201.103@tcp failed: rc = -19
[ 2895.815433] Lustre: lustre-MDT0000-mdc-ffff96fd057df000: Connection to lustre-MDT0000 (at 192.168.201.103@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2895.845958] LustreError: Skipped 1 previous similar message
[ 2895.874021] Lustre: Skipped 5 previous similar messages
[ 2916.895359] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 192.168.201.103@tcp) was lost; in progress operations using this service will fail
[ 2927.236044] Lustre: Evicted from MGS (at 192.168.201.103@tcp) after server handle changed from 0xaa643b29f94c1d2c to 0xaa643b29f94d5c4d
[ 2927.285651] Lustre: MGC192.168.201.103@tcp: Connection restored to 192.168.201.103@tcp (at 192.168.201.103@tcp)
[ 2927.302766] Lustre: Skipped 11 previous similar messages
[ 2953.479074] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2955.397762] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2987.832719] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 18:04:43 (1781820283)
[ 3012.639250] Lustre: 2354:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781820294/real 1781820294]  req@ffff96fd086ec700 x1868370984861184/t0(0) o400->MGC192.168.201.103@tcp@192.168.201.103@tcp:26/25 lens 224/224 e 0 to 1 dl 1781820310 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3012.707067] Lustre: 2354:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 12 previous similar messages
[ 3012.734305] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 192.168.201.103@tcp) was lost; in progress operations using this service will fail
[ 3023.365511] Lustre: Evicted from MGS (at 192.168.201.103@tcp) after server handle changed from 0xaa643b29f94d5c4d to 0xaa643b29f94df7f8
[ 3024.145512] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 3074.304095] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 3081.909930] LustreError: lustre-MDT0000-mdc-ffff96fd057df000: operation ldlm_enqueue to node 192.168.201.103@tcp failed: rc = -19
[ 3081.924194] LustreError: Skipped 2 previous similar messages
[ 3124.828773] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 3160.550226] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 192.168.201.103@tcp) was lost; in progress operations using this service will fail
[ 3160.586275] LustreError: Skipped 2 previous similar messages
[ 3160.609606] Lustre: Evicted from MGS (at 192.168.201.103@tcp) after server handle changed from 0xaa643b29f94e0c26 to 0xaa643b29f94e5583
[ 3160.623250] Lustre: Skipped 2 previous similar messages
[ 3162.293341] Lustre: 15888:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.201.103@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 3185.611262] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 3254.859970] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 3313.154291] Lustre: 15888:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.201.103@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 3325.067285] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 3362.042487] LustreError: lustre-MDT0000-mdc-ffff96fd057df000: operation ldlm_enqueue to node 192.168.201.103@tcp failed: rc = -19
[ 3362.081526] LustreError: Skipped 5 previous similar messages
[ 3431.408234] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 18:12:05 (1781820725)
[ 3503.764162] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3506.611903] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3780.770065] Lustre: lustre-OST0000-osc-ffff96fd057df000: Connection to lustre-OST0000 (at 192.168.201.103@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3780.850179] Lustre: Skipped 7 previous similar messages
[ 3811.628050] Lustre: lustre-OST0000-osc-ffff96fd057df000: Connection restored to 192.168.201.103@tcp (at 192.168.201.103@tcp)
[ 3811.687424] Lustre: Skipped 14 previous similar messages
[ 3838.526858] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3841.491747] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4111.746396] LustreError: lustre-OST0000-osc-ffff96fd057df000: operation ldlm_enqueue to node 192.168.201.103@tcp failed: rc = -107
[ 4111.777722] LustreError: Skipped 2 previous similar messages
[ 4170.159379] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4173.688094] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4408.813062] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 18:28:24 (1781821704)
[ 4427.231171] Lustre: 50333:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781821708/real 1781821708]  req@ffff96fd09bb9f80 x1868370996504320/t0(0) o101->lustre-MDT0000-mdc-ffff96fd057df000@192.168.201.103@tcp:12/10 lens 576/1152 e 0 to 1 dl 1781821724 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 4427.269595] Lustre: 50333:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
[ 4427.274146] Lustre: lustre-MDT0000-mdc-ffff96fd057df000: Connection to lustre-MDT0000 (at 192.168.201.103@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4427.287595] Lustre: Skipped 1 previous similar message
[ 4427.340893] Lustre: lustre-MDT0000-mdc-ffff96fd057df000: Connection restored to 192.168.201.103@tcp (at 192.168.201.103@tcp)
[ 4427.358096] Lustre: Skipped 1 previous similar message
[ 4441.296449] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 18:28:56 (1781821736)
[ 4472.535739] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 18:29:27 (1781821767)
[ 4502.008959] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 18:29:57 (1781821797)
[ 4502.601488] Lustre: Mounted lustre-client
[ 4534.239428] Lustre: 2355:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781821815/real 1781821815]  req@ffff96fd0a229880 x1868370996525696/t0(0) o400->MGC192.168.201.103@tcp@192.168.201.103@tcp:26/25 lens 224/224 e 0 to 1 dl 1781821831 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4534.276706] Lustre: 2355:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 4534.288431] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 192.168.201.103@tcp) was lost; in progress operations using this service will fail
[ 4534.310253] LustreError: Skipped 3 previous similar messages
[ 4544.522373] Lustre: Evicted from MGS (at 192.168.201.103@tcp) after server handle changed from 0xaa643b29f94fa41e to 0xaa643b29f95b917c
[ 4544.533639] Lustre: Skipped 3 previous similar messages
[ 4563.634780] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4566.017803] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4568.990920] Lustre: Unmounted lustre-client
[ 4579.564201] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 18:31:14 (1781821874)
[ 4691.935253] Lustre: 2356:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781821973/real 1781821973]  req@ffff96fd08b90700 x1868370996562176/t0(0) o4->lustre-OST0000-osc-ffff96fd057df000@192.168.201.103@tcp:6/4 lens 488/448 e 0 to 1 dl 1781821989 ref 2 fl Rpc:XQr/602/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4691.991115] Lustre: 2356:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 57 previous similar messages
[ 4888.989591] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 18:36:24 (1781822184)
[ 4942.541968] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 18:37:17 (1781822237)
[ 4947.682323] Lustre: Unmounted lustre-client
[ 4978.049593] Lustre: Mounted lustre-client
[ 4987.114400] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 18:38:02 (1781822282)
[ 4987.729180] LustreError: 56257:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4988.801337] LustreError: 56257:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4988.975754] Lustre: *** cfs_fail_loc=305, val=0***
[ 5013.712142] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 18:38:29 (1781822309)
[ 5014.172616] Lustre: Mounted lustre-client
[ 5018.325187] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 5028.665314] Lustre: Unmounted lustre-client
[ 5041.318843] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 18:38:56 (1781822336)
[ 5202.967372] LustreError: lustre-MDT0000-mdc-ffff96fd09ad9800: operation mds_reint to node 192.168.201.103@tcp failed: rc = -19
[ 5202.987550] Lustre: lustre-MDT0000-mdc-ffff96fd09ad9800: Connection to lustre-MDT0000 (at 192.168.201.103@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5203.016307] Lustre: Skipped 23 previous similar messages
[ 5220.327563] Lustre: 2355:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781822501/real 1781822501]  req@ffff96fd04682300 x1868370999233664/t0(0) o400->MGC192.168.201.103@tcp@192.168.201.103@tcp:26/25 lens 224/224 e 0 to 1 dl 1781822517 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5220.390047] Lustre: 2355:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 93 previous similar messages
[ 5220.412351] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 192.168.201.103@tcp) was lost; in progress operations using this service will fail
[ 5229.632385] Lustre: Evicted from MGS (at 192.168.201.103@tcp) after server handle changed from 0xaa643b29f95bac79 to 0xaa643b29f95e5b28
[ 5229.660450] Lustre: MGC192.168.201.103@tcp: Connection restored to 192.168.201.103@tcp (at 192.168.201.103@tcp)
[ 5229.695329] Lustre: Skipped 24 previous similar messages
[ 5512.050290] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 18:46:47 (1781822807)
[ 5519.248938] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5541.365595] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 192.168.201.103@tcp) was lost; in progress operations using this service will fail
[ 5541.397393] Lustre: Evicted from MGS (at 192.168.201.103@tcp) after server handle changed from 0xaa643b29f95e5b28 to 0xaa643b29f9633076
[ 5541.461450] LustreError: lustre-MDT0000-mdc-ffff96fd09ad9800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5568.056161] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 18:47:42 (1781822862)
[ 5569.116616] Lustre: Mounted lustre-client
[ 5577.778616] LustreError: lustre-OST0000-osc-ffff96fd09ad9800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5591.104407] Lustre: Unmounted lustre-client
[ 5601.095491] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 18:48:16 (1781822896)
[ 5609.078570] LustreError: lustre-MDT0000-mdc-ffff96fd09ad9800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5609.118691] LustreError: 60689:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 5622.508402] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 18:48:37 (1781822917)
[ 5622.740955] LustreError: 61373:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 5627.384150] LustreError: lustre-MDT0000-mdc-ffff96fd09ad9800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5627.422463] LustreError: 61397:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 5627.448427] LustreError: 61373:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=307
[ 5627.488801] LustreError: 61373:0:(import.c:719:ptlrpc_connect_import_locked()) already connecting
[ 5627.541581] LustreError: 61402:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5628.833846] LustreError: 61408:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5628.848700] LustreError: 61408:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5631.300531] LustreError: 61430:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5631.323101] LustreError: 61430:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5636.273967] LustreError: 61475:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5636.290307] LustreError: 61475:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 5651.819796] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 18:49:06 (1781822946)
[ 5709.007542] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5711.691500] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5726.425712] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 18:50:21 (1781823021)
[ 5782.022913] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5785.588845] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5808.797207] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 18:51:43 (1781823103)
[ 5908.578193] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5912.496121] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5928.226894] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 18:53:41 (1781823221)
[ 5999.883789] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 6002.388663] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6012.871936] Lustre: Unmounted lustre-client
[ 6114.633422] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 6119.381805] Lustre: Mounted lustre-client
[ 6135.997808] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 18:57:09 (1781823429)
[ 6141.240974] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 6145.590509] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 18:57:19 (1781823439)
[ 6218.947659] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 18:58:33 (1781823513)
[ 6221.629603] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 6226.326875] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 18:58:39 (1781823519)
[ 6229.130947] Lustre: *** cfs_fail_loc=805, val=0***
[ 6229.257297] Lustre: Mounted lustre-client
[ 6242.920547] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6266.336693] Lustre: 2356:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781823547/real 1781823547]  req@ffff96fd043b9f80 x1868371001320960/t0(0) o400->MGC192.168.201.103@tcp@192.168.201.103@tcp:26/25 lens 224/224 e 0 to 1 dl 1781823563 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6266.394526] Lustre: 2356:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 6266.434811] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 192.168.201.103@tcp) was lost; in progress operations using this service will fail
[ 6266.489919] Lustre: lustre-MDT0000-mdc-ffff96fd18907800: Connection to lustre-MDT0000 (at 192.168.201.103@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6266.576298] Lustre: Skipped 8 previous similar messages
[ 6291.246118] Lustre: Evicted from MGS (at 192.168.201.103@tcp) after server handle changed from 0xaa643b29f9633a7f to 0xaa643b29f9633ca8
[ 6291.314328] Lustre: MGC192.168.201.103@tcp: Connection restored to 192.168.201.103@tcp (at 192.168.201.103@tcp)
[ 6291.345363] Lustre: Skipped 9 previous similar messages
[ 6293.198806] Lustre: 68006:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.201.103@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6320.846184] Lustre: Unmounted lustre-client
[ 6339.647765] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 19:00:33 (1781823633)
[ 6369.251557] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 192.168.201.103@tcp) was lost; in progress operations using this service will fail
[ 6393.851398] Lustre: Evicted from MGS (at 192.168.201.103@tcp) after server handle changed from 0xaa643b29f9633ca8 to 0xaa643b29f9634307
[ 6404.243593] LustreError: 2353:0:(client.c:3447:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff96fd0a0e8380 x1868371001334912/t90194313220(90194313220) o101->lustre-MDT0000-mdc-ffff96fd09ab3800@192.168.201.103@tcp:12/10 lens 664/608 e 0 to 0 dl 1781823717 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 6429.447722] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6432.761571] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6448.480429] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 19:02:23 (1781823743)
[ 6450.807907] LustreError: lustre-OST0000-osc-ffff96fd09ab3800: operation ost_write to node 192.168.201.103@tcp failed: rc = -107
[ 6450.833158] LustreError: Skipped 3 previous similar messages
[ 6450.868273] LustreError: lustre-OST0000-osc-ffff96fd09ab3800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6450.937481] Lustre: 2354:0:(llite_lib.c:4198:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.103@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 6451.062193] LustreError: 72557:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-OST0000-osc-ffff96fd09ab3800: namespace resource [0x240000400:0x2b02:0x0].0x0 (ffff96fd05c9ae00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6469.953746] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 19:02:43 (1781823763)
[ 6473.743434] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 6478.180049] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 19:02:52 (1781823772)
[ 6481.880344] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 6486.342805] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 19:03:00 (1781823780)
[ 6490.219733] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 6494.272791] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 19:03:08 (1781823788)
[ 6497.876246] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 6501.726231] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 19:03:16 (1781823796)
[ 6505.175771] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 6509.000987] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 19:03:23 (1781823803)
[ 6512.969738] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 6518.204371] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 19:03:31 (1781823811)
[ 6522.298259] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 6526.461300] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 19:03:40 (1781823820)
[ 6529.651401] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 6533.528864] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 19:03:47 (1781823827)
[ 6537.010388] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 6540.727579] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 19:03:55 (1781823835)
[ 6544.035126] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 6548.333175] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 19:04:02 (1781823842)
[ 6552.625906] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 6556.430213] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 19:04:10 (1781823850)
[ 6562.767285] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 6566.855613] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 19:04:21 (1781823861)
[ 6592.483195] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 192.168.201.103@tcp) was lost; in progress operations using this service will fail
[ 6617.008197] Lustre: Evicted from MGS (at 192.168.201.103@tcp) after server handle changed from 0xaa643b29f9634307 to 0xaa643b29f9634990
[ 6644.055270] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 19:05:37 (1781823937)
[ 6687.906210] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 19:06:21 (1781823981)
[ 6688.858912] Lustre: *** cfs_fail_loc=51b, val=3***
[ 6709.304365] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 19:06:42 (1781824002)
[ 6710.981139] Lustre: *** cfs_fail_loc=51b, val=4***
[ 6718.817582] Lustre: DEBUG MARKER: recovery-small test_115b: @@@@@@ FAIL: dd success