[ 1902.908304] LustreError: lustre-OST0000-osc-ffff92075046d000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1902.932352] Lustre: 2361:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.104@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1909.179669] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 00:59:28 (1787374768)
[ 1911.514214] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1913.249806] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 00:59:32 (1787374772)
[ 1915.458753] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1917.462599] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 00:59:36 (1787374776)
[ 1921.249942] LustreError: lustre-MDT0000-mdc-ffff92075046d000: operation ldlm_enqueue to node 192.168.201.104@tcp failed: rc = -19
[ 1921.261076] LustreError: Skipped 1 previous similar message
[ 1941.985516] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 192.168.201.104@tcp) was lost; in progress operations using this service will fail
[ 1942.017798] Lustre: Evicted from MGS (at 192.168.201.104@tcp) after server handle changed from 0x9d9168dda0cf9a0e to 0x9d9168dda0cfb2b1
[ 1943.291489] Lustre: 15933:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.104@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2052.891759] LustreError: lustre-MDT0000-mdc-ffff92075046d000: operation ldlm_enqueue to node 192.168.201.104@tcp failed: rc = -19
[ 2068.962750] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 192.168.201.104@tcp) was lost; in progress operations using this service will fail
[ 2079.220918] Lustre: Evicted from MGS (at 192.168.201.104@tcp) after server handle changed from 0x9d9168dda0cfb2b1 to 0x9d9168dda0d193ce
[ 2095.171935] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 01:02:33 (1787374953)
[ 2095.739735] Lustre: *** cfs_fail_loc=305, val=0***
[ 2095.743885] Lustre: Skipped 2 previous similar messages
[ 2140.640633] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 192.168.201.104@tcp) was lost; in progress operations using this service will fail
[ 2140.664901] Lustre: Evicted from MGS (at 192.168.201.104@tcp) after server handle changed from 0x9d9168dda0d193ce to 0x9d9168dda0d197f6
[ 2154.778592] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2156.333422] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2165.914567] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 01:03:44 (1787375024)
[ 2180.595282] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 192.168.201.104@tcp) was lost; in progress operations using this service will fail
[ 2180.639874] Lustre: Evicted from MGS (at 192.168.201.104@tcp) after server handle changed from 0x9d9168dda0d197f6 to 0x9d9168dda0d19cf7
[ 2180.727296] LustreError: lustre-MDT0000-mdc-ffff92075046d000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2201.980943] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 01:04:20 (1787375060)
[ 2206.194197] Lustre: lustre-OST0000-osc-ffff92075046d000: Connection to lustre-OST0000 (at 192.168.201.104@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2206.214948] Lustre: Skipped 12 previous similar messages
[ 2213.902275] LustreError: lustre-OST0000-osc-ffff92075046d000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2213.934186] Lustre: lustre-OST0000-osc-ffff92075046d000: Connection restored to 192.168.201.104@tcp (at 192.168.201.104@tcp)
[ 2213.944692] Lustre: Skipped 16 previous similar messages
[ 2233.281974] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 01:04:51 (1787375091)
[ 2248.248958] LustreError: lustre-MDT0000-mdc-ffff92075046d000: operation mds_close to node 192.168.201.104@tcp failed: rc = -19
[ 2248.260256] LustreError: Skipped 2 previous similar messages
[ 2267.645308] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 192.168.201.104@tcp) was lost; in progress operations using this service will fail
[ 2267.686262] Lustre: Evicted from MGS (at 192.168.201.104@tcp) after server handle changed from 0x9d9168dda0d19cf7 to 0x9d9168dda0d1e359
[ 2285.212828] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2287.903267] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2375.164079] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 192.168.201.104@tcp) was lost; in progress operations using this service will fail
[ 2375.204242] Lustre: Evicted from MGS (at 192.168.201.104@tcp) after server handle changed from 0x9d9168dda0d1e359 to 0x9d9168dda0d3d70e
[ 2393.019023] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2395.405080] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2461.934907] LustreError: lustre-MDT0000-mdc-ffff92075046d000: operation ldlm_enqueue to node 192.168.201.104@tcp failed: rc = -19
[ 2461.953858] LustreError: Skipped 2 previous similar messages
[ 2482.672838] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 192.168.201.104@tcp) was lost; in progress operations using this service will fail
[ 2482.707361] Lustre: Evicted from MGS (at 192.168.201.104@tcp) after server handle changed from 0x9d9168dda0d3d70e to 0x9d9168dda0d5bda3
[ 2496.674415] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2499.261284] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2528.562636] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 01:09:47 (1787375387)
[ 2553.347116] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 192.168.201.104@tcp) was lost; in progress operations using this service will fail
[ 2553.403147] Lustre: Evicted from MGS (at 192.168.201.104@tcp) after server handle changed from 0x9d9168dda0d5bda3 to 0x9d9168dda0d6ac8a
[ 2565.604538] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2597.293229] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2632.603321] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2664.928440] Lustre: 2364:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787375509/real 1787375509]  req@ffff92074993ca80 x1874196412081152/t0(0) o400->MGC192.168.201.104@tcp@192.168.201.104@tcp:26/25 lens 224/224 e 0 to 1 dl 1787375525 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2664.961798] Lustre: 2364:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 12 previous similar messages
[ 2682.336994] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2726.379044] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 192.168.201.104@tcp) was lost; in progress operations using this service will fail
[ 2726.390128] LustreError: Skipped 3 previous similar messages
[ 2736.617687] Lustre: Evicted from MGS (at 192.168.201.104@tcp) after server handle changed from 0x9d9168dda0d7f22d to 0x9d9168dda0d86fa1
[ 2736.630575] Lustre: Skipped 3 previous similar messages
[ 2743.024959] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2770.630246] LustreError: lustre-MDT0000-mdc-ffff92075046d000: operation mds_reint to node 192.168.201.104@tcp failed: rc = -19
[ 2770.648015] LustreError: Skipped 6 previous similar messages
[ 2798.750042] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2832.001388] Lustre: lustre-MDT0000-mdc-ffff92075046d000: Connection to lustre-MDT0000 (at 192.168.201.104@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2832.015862] Lustre: Skipped 9 previous similar messages
[ 2851.311532] Lustre: MGC192.168.201.104@tcp: Connection restored to 192.168.201.104@tcp (at 192.168.201.104@tcp)
[ 2851.317033] Lustre: Skipped 18 previous similar messages
[ 2853.366234] Lustre: 15933:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.104@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2891.227841] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 01:15:50 (1787375750)
[ 2943.347655] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 2945.687861] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3278.993807] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3281.361237] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3568.350939] LustreError: lustre-OST0000-osc-ffff92075046d000: operation ldlm_enqueue to node 192.168.201.104@tcp failed: rc = -107
[ 3568.362565] LustreError: Skipped 4 previous similar messages
[ 3568.366081] Lustre: lustre-OST0000-osc-ffff92075046d000: Connection to lustre-OST0000 (at 192.168.201.104@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3568.381540] Lustre: Skipped 2 previous similar messages
[ 3593.793819] Lustre: lustre-OST0000-osc-ffff92075046d000: Connection restored to 192.168.201.104@tcp (at 192.168.201.104@tcp)
[ 3593.828584] Lustre: Skipped 3 previous similar messages
[ 3615.009777] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3617.249719] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3863.045647] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 01:32:01 (1787376721)
[ 3879.904458] Lustre: 50560:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787376724/real 1787376724]  req@ffff920750830380 x1874196431635072/t0(0) o101->lustre-MDT0000-mdc-ffff92075046d000@192.168.201.104@tcp:12/10 lens 576/1152 e 0 to 1 dl 1787376740 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3879.930112] Lustre: 50560:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 3888.973735] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 01:32:27 (1787376747)
[ 3914.676629] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 01:32:53 (1787376773)
[ 3940.190937] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 01:33:19 (1787376799)
[ 3940.659615] Lustre: Mounted lustre-client - version 2.17.57_45_gb800aec
[ 3972.585129] Lustre: 2364:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787376816/real 1787376816]  req@ffff920750949f80 x1874196431654272/t0(0) o400->lustre-MDT0000-mdc-ffff92075046d000@192.168.201.104@tcp:12/10 lens 224/224 e 0 to 1 dl 1787376832 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3972.627099] Lustre: 2364:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 3972.647526] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 192.168.201.104@tcp) was lost; in progress operations using this service will fail
[ 3972.660901] LustreError: Skipped 2 previous similar messages
[ 3972.680342] Lustre: Evicted from MGS (at 192.168.201.104@tcp) after server handle changed from 0x9d9168dda0d9c478 to 0x9d9168dda0eef261
[ 3972.696572] Lustre: Skipped 2 previous similar messages
[ 3987.173275] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 3988.967119] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3991.388375] Lustre: Unmounted lustre-client
[ 3998.129461] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 01:34:17 (1787376857)
[ 4134.368150] Lustre: 2362:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787376978/real 1787376978]  req@ffff920750833100 x1874196431686784/t0(0) o4->lustre-OST0000-osc-ffff92075046d000@192.168.201.104@tcp:6/4 lens 488/448 e 0 to 1 dl 1787376994 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4134.368150] Lustre: 2361:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787376978/real 1787376978]  req@ffff920750830700 x1874196431687936/t0(0) o4->lustre-OST0000-osc-ffff92075046d000@192.168.201.104@tcp:6/4 lens 488/448 e 0 to 1 dl 1787376994 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4134.368163] Lustre: 2362:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 28 previous similar messages
[ 4134.387650] Lustre: 2361:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 32 previous similar messages
[ 4183.136472] Lustre: lustre-OST0000-osc-ffff92075046d000: Connection to lustre-OST0000 (at 192.168.201.104@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4183.149140] Lustre: Skipped 15 previous similar messages
[ 4198.891419] Lustre: lustre-OST0000-osc-ffff92075046d000: Connection restored to 192.168.201.104@tcp (at 192.168.201.104@tcp)
[ 4198.897628] Lustre: Skipped 17 previous similar messages
[ 4290.485311] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 01:39:09 (1787377149)
[ 4339.876342] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 01:39:58 (1787377198)
[ 4343.310872] Lustre: Unmounted lustre-client
[ 4368.488777] Lustre: Mounted lustre-client - version 2.17.57_45_gb800aec
[ 4375.516454] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 01:40:34 (1787377234)
[ 4375.877560] LustreError: 56481:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4376.934212] LustreError: 56481:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4377.126948] Lustre: *** cfs_fail_loc=305, val=0***
[ 4401.570034] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 01:41:00 (1787377260)
[ 4402.223743] Lustre: Mounted lustre-client - version 2.17.57_45_gb800aec
[ 4405.198542] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4415.502084] Lustre: Unmounted lustre-client
[ 4423.250074] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 01:41:21 (1787377281)
[ 4500.724858] LustreError: lustre-MDT0000-mdc-ffff92074922c000: operation mds_reint to node 192.168.201.104@tcp failed: rc = -19
[ 4520.928251] Lustre: 2364:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787377365/real 1787377365]  req@ffff9207443ff800 x1874196434356864/t0(0) o400->MGC192.168.201.104@tcp@192.168.201.104@tcp:26/25 lens 224/224 e 0 to 1 dl 1787377381 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4520.956409] Lustre: 2364:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 61 previous similar messages
[ 4520.960236] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 192.168.201.104@tcp) was lost; in progress operations using this service will fail
[ 4520.974631] Lustre: Evicted from MGS (at 192.168.201.104@tcp) after server handle changed from 0x9d9168dda0ef08b8 to 0x9d9168dda0f1c043
[ 4683.159229] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 01:45:41 (1787377541)
[ 4688.966306] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4704.237149] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 192.168.201.104@tcp) was lost; in progress operations using this service will fail
[ 4704.265192] Lustre: Evicted from MGS (at 192.168.201.104@tcp) after server handle changed from 0x9d9168dda0f1c043 to 0x9d9168dda0f6868e
[ 4704.391546] LustreError: lustre-MDT0000-mdc-ffff92074922c000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4725.653874] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 01:46:23 (1787377583)
[ 4726.260908] Lustre: Mounted lustre-client - version 2.17.57_45_gb800aec
[ 4733.741524] LustreError: lustre-OST0000-osc-ffff92074922c000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4747.700502] Lustre: Unmounted lustre-client
[ 4755.824783] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 01:46:54 (1787377614)
[ 4762.472525] LustreError: lustre-MDT0000-mdc-ffff92074922c000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4762.488930] LustreError: 60871:0:(file.c:6170:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4771.480482] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 01:47:10 (1787377630)
[ 4772.013314] LustreError: 61576:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4777.440311] LustreError: 61576:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=0
[ 4777.451995] LustreError: 61576:0:(import.c:711:ptlrpc_connect_import_locked()) already connected
[ 4778.373229] LustreError: lustre-MDT0000-mdc-ffff92074922c000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4778.410617] LustreError: 61593:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4778.500240] LustreError: 61598:0:(file.c:6170:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4779.722708] LustreError: 61604:0:(file.c:6170:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4779.729654] LustreError: 61604:0:(file.c:6170:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4782.074143] LustreError: 61627:0:(file.c:6170:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4782.094278] LustreError: 61627:0:(file.c:6170:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4786.794503] LustreError: 61673:0:(file.c:6170:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4786.808147] LustreError: 61673:0:(file.c:6170:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4797.690951] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 01:47:36 (1787377656)
[ 4804.079667] Lustre: lustre-OST0000-osc-ffff92074922c000: Connection to lustre-OST0000 (at 192.168.201.104@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4804.087040] Lustre: Skipped 17 previous similar messages
[ 4841.025139] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4842.661301] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4855.298313] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 01:48:33 (1787377713)
[ 4878.488104] Lustre: lustre-OST0000-osc-ffff92074922c000: Connection restored to 192.168.201.104@tcp (at 192.168.201.104@tcp)
[ 4878.495295] Lustre: Skipped 17 previous similar messages
[ 4894.082893] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4895.855980] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4908.596213] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 01:49:26 (1787377766)
[ 4977.603903] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4979.929350] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4989.168927] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 01:50:47 (1787377847)
[ 5030.711527] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5032.443335] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5039.320593] Lustre: Unmounted lustre-client
[ 5092.976921] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5095.998463] Lustre: Mounted lustre-client - version 2.17.57_45_gb800aec
[ 5102.415282] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 01:52:41 (1787377961)
[ 5104.872720] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5106.456797] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 01:52:45 (1787377965)
[ 5143.430605] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 01:53:21 (1787378001)
[ 5145.750226] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5147.729760] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 01:53:26 (1787378006)
[ 5149.217622] Lustre: *** cfs_fail_loc=805, val=0***
[ 5149.294969] Lustre: Mounted lustre-client - version 2.17.57_45_gb800aec
[ 5155.216504] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5176.293850] Lustre: 2362:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787378020/real 1787378020]  req@ffff920744249c00 x1874196436409088/t0(0) o400->lustre-MDT0000-mdc-ffff92075049f000@192.168.201.104@tcp:12/10 lens 224/224 e 0 to 1 dl 1787378036 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5176.319162] Lustre: 2362:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5176.334163] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 192.168.201.104@tcp) was lost; in progress operations using this service will fail
[ 5186.579915] Lustre: Evicted from MGS (at 192.168.201.104@tcp) after server handle changed from 0x9d9168dda0f69090 to 0x9d9168dda0f693f4
[ 5194.254132] Lustre: Unmounted lustre-client
[ 5202.358131] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 01:54:20 (1787378060)
[ 5235.504039] Lustre: 68243:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.104@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 5243.966349] LustreError: 2360:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff92074424b100 x1874196436420864/t94489280516(94489280516) o101->lustre-MDT0000-mdc-ffff92075049f000@192.168.201.104@tcp:12/10 lens 664/608 e 0 to 0 dl 1787378120 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5250.278966] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5252.103205] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5260.442286] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 01:55:19 (1787378119)
[ 5262.069583] LustreError: lustre-OST0000-osc-ffff92075049f000: operation ost_write to node 192.168.201.104@tcp failed: rc = -107
[ 5262.087387] LustreError: Skipped 3 previous similar messages
[ 5262.143153] LustreError: lustre-OST0000-osc-ffff92075049f000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5262.162147] Lustre: 2363:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.104@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5262.215463] LustreError: 72783:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff92075049f000: namespace resource [0x240000400:0x4062:0x0].0x0 (ffff92074b342900) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5270.189744] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 01:55:28 (1787378128)
[ 5271.711570] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5273.418340] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 01:55:32 (1787378132)
[ 5274.981962] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5276.364708] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 01:55:35 (1787378135)
[ 5278.057579] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5279.886317] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 01:55:38 (1787378138)
[ 5281.607335] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5283.346690] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 01:55:42 (1787378142)
[ 5284.867174] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5286.573396] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 01:55:45 (1787378145)
[ 5288.048570] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5289.811453] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 01:55:48 (1787378148)
[ 5291.754308] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5293.523335] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 01:55:52 (1787378152)
[ 5295.008599] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5296.956983] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 01:55:55 (1787378155)
[ 5298.369589] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5300.078575] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 01:55:58 (1787378158)
[ 5301.724603] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5303.528143] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 01:56:02 (1787378162)
[ 5305.242692] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5307.356311] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 01:56:05 (1787378165)
[ 5310.027467] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5311.686407] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 01:56:10 (1787378170)
[ 5347.191913] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 01:56:45 (1787378205)
[ 5377.475103] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 01:57:16 (1787378236)
[ 5378.133842] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5388.727911] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 01:57:27 (1787378247)
[ 5389.437985] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5400.816409] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 01:57:39 (1787378259)
[ 5401.169509] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5410.094373] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 01:57:48 (1787378268)
[ 5410.594048] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5420.212532] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 01:57:59 (1787378279)
[ 5420.571645] Lustre: *** cfs_fail_loc=510, val=3***
[ 5428.685803] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 01:58:07 (1787378287)
[ 5429.128492] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5439.906800] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 01:58:18 (1787378298)
[ 5440.294560] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5498.848579] Lustre: lustre-OST0000-osc-ffff92075049f000: Connection to lustre-OST0000 (at 192.168.201.104@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5498.861892] Lustre: Skipped 9 previous similar messages
[ 5505.603637] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 01:59:24 (1787378364)
[ 5505.953907] LustreError: 82992:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5508.892893] LustreError: lustre-MDT0000-mdc-ffff92075049f000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5508.913440] LustreError: 83007:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff92075049f000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff920744990f00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5508.960464] Lustre: lustre-MDT0000-mdc-ffff92075049f000: Connection restored to 192.168.201.104@tcp (at 192.168.201.104@tcp)
[ 5508.979609] Lustre: Skipped 10 previous similar messages
[ 5510.057542] LustreError: 82992:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5512.214159] LustreError: 83022:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5515.247892] LustreError: lustre-MDT0000-mdc-ffff92075049f000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5515.268628] LustreError: 83037:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff92075049f000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff920744990000) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5515.296247] LustreError: 83037:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5516.304483] LustreError: 83022:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5523.196309] LustreError: lustre-MDT0000-mdc-ffff92075049f000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5523.222413] LustreError: 83065:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff92075049f000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff920745d2de00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5523.233609] LustreError: 83065:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5523.697057] LustreError: 83078:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5526.763213] LustreError: lustre-MDT0000-mdc-ffff92075049f000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5526.879886] LustreError: 83098:0:(file.c:6170:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5526.899819] LustreError: 83098:0:(file.c:6170:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5527.768149] LustreError: 83078:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5527.776596] LustreError: 83078:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff92075049f000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5528.061471] LustreError: 83104:0:(file.c:6170:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5528.074550] LustreError: 83104:0:(file.c:6170:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5530.440165] LustreError: 83126:0:(file.c:6170:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5530.461270] LustreError: 83126:0:(file.c:6170:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5534.018434] LustreError: 83153:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5537.075258] LustreError: lustre-MDT0000-mdc-ffff92075049f000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5537.100872] LustreError: 83168:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff92075049f000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff920745d88c00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5538.104171] LustreError: 83153:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5545.097411] LustreError: lustre-MDT0000-mdc-ffff92075049f000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5545.630383] LustreError: 83211:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5548.751566] LustreError: lustre-MDT0000-mdc-ffff92075049f000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5548.772915] LustreError: 83226:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff92075049f000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff920744990500) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5548.814306] LustreError: 83226:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5549.732248] LustreError: 83211:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5552.839773] LustreError: lustre-MDT0000-mdc-ffff92075049f000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5552.927218] LustreError: 83259:0:(file.c:6170:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5552.937784] LustreError: 83259:0:(file.c:6170:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5553.920731] LustreError: 83239:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff92075049f000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5562.193326] LustreError: 83311:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5562.207561] LustreError: 83311:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5565.667809] LustreError: lustre-MDT0000-mdc-ffff92075049f000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5565.718906] LustreError: 83333:0:(file.c:6170:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5565.727400] LustreError: 83333:0:(file.c:6170:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5566.288759] LustreError: 83311:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5566.296840] LustreError: 83311:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5566.308379] LustreError: 83311:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff92075049f000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5576.541398] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 02:00:35 (1787378435)
[ 5595.560251] LustreError: 68240:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5595.570205] LustreError: 68240:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5599.640161] LustreError: 68240:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5599.650655] LustreError: 68240:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5606.501695] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 02:01:05 (1787378465)
[ 5655.543344] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 02:01:54 (1787378514)
[ 5718.892345] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 02:02:58 (1787378578)
[ 5753.769422] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 02:03:32 (1787378612)
[ 5754.306238] Lustre: Mounted lustre-client - version 2.17.57_45_gb800aec
[ 5877.409228] Lustre: Unmounted lustre-client
[ 5883.299648] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 02:05:42 (1787378742)
[ 5883.812951] LustreError: 87646:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5885.972448] LustreError: lustre-OST0000-osc-ffff92075049f000: operation ost_statfs to node 192.168.201.104@tcp failed: rc = -107
[ 5885.980673] LustreError: Skipped 9 previous similar messages
[ 5885.988482] LustreError: lustre-OST0000-osc-ffff92075049f000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5885.994136] LustreError: 87733:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff92075049f000: namespace resource [0x240000400:0x408d:0x0].0x0 (ffff920744990200) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5886.006454] LustreError: 87733:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5886.232479] LustreError: 87646:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5886.242840] Lustre: 2361:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.104@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5891.382693] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 02:05:50 (1787378750)
[ 5949.408246] Lustre: 88314:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787378754/real 1787378754]  req@ffff920750543480 x1874196436590592/t0(0) o101->lustre-MDT0000-mdc-ffff92075049f000@192.168.201.104@tcp:12/10 lens 328/344 e 0 to 1 dl 1787378809 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 5949.442141] Lustre: 88314:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 24 previous similar messages
[ 5956.831690] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 02:06:55 (1787378815)
[ 5958.454900] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 5959.923684] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 02:06:59 (1787378819)
[ 6022.892521] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6024.346942] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 02:08:03 (1787378883)
[ 6082.296789] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 02:09:01 (1787378941)
[ 6085.271697] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6087.118553] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 02:09:05 (1787378945)
[ 6088.746624] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6090.635743] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 02:09:09 (1787378949)
[ 6124.903841] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 02:09:43 (1787378983)
[ 6135.741759] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6162.419724] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 192.168.201.104@tcp) was lost; in progress operations using this service will fail
[ 6162.439091] LustreError: Skipped 2 previous similar messages
[ 6162.453661] Lustre: Evicted from MGS (at 192.168.201.104@tcp) after server handle changed from 0x9d9168dda0f69f0e to 0x9d9168dda0f6b741
[ 6162.459127] Lustre: Skipped 2 previous similar messages
[ 6162.463495] Lustre: MGC192.168.201.104@tcp: Connection restored to 192.168.201.104@tcp (at 192.168.201.104@tcp)
[ 6162.470619] Lustre: Skipped 15 previous similar messages
[ 6164.405326] Lustre: 68243:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.104@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6169.568372] Lustre: lustre-MDT0000-mdc-ffff92075049f000: Connection to lustre-MDT0000 (at 192.168.201.104@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6169.586156] Lustre: Skipped 16 previous similar messages
[ 6184.306836] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6185.733335] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6195.489424] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 02:10:54 (1787379054)
[ 6197.891323] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6199.397128] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 02:10:58 (1787379058)
[ 6214.657310] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 192.168.201.104@tcp) was lost; in progress operations using this service will fail
[ 6214.697641] Lustre: Evicted from MGS (at 192.168.201.104@tcp) after server handle changed from 0x9d9168dda0f6b741 to 0x9d9168dda0f6bbfc
[ 6230.002275] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 02:11:28 (1787379088)
[ 6275.914945] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 02:12:14 (1787379134)
[ 6342.564210] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6347.243387] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6434.592636] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 192.168.201.104@tcp) was lost; in progress operations using this service will fail
[ 6434.612384] LustreError: Skipped 1 previous similar message
[ 6434.630488] Lustre: Evicted from MGS (at 192.168.201.104@tcp) after server handle changed from 0x9d9168dda0f6bf83 to 0x9d9168dda0f6cc56
[ 6434.637751] Lustre: Skipped 1 previous similar message
[ 6436.038638] Lustre: 68243:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.104@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6458.899662] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6460.900509] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6487.217282] Lustre: 68243:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.104@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6510.538707] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6512.607611] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6651.304702] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 02:18:30 (1787379510)
[ 6663.267074] LustreError: lustre-OST0000-osc-ffff92075049f000: operation ost_setattr to node 192.168.201.104@tcp failed: rc = -19
[ 6663.275634] LustreError: Skipped 18 previous similar messages
[ 6722.632605] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6727.736249] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6992.461419] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 02:24:11 (1787379851)
[ 7144.443092] Lustre: lustre-MDT0000-mdc-ffff92075049f000: Connection to lustre-MDT0000 (at 192.168.201.104@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7144.475506] Lustre: Skipped 6 previous similar messages
[ 7144.483825] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 192.168.201.104@tcp) was lost; in progress operations using this service will fail
[ 7144.502031] LustreError: Skipped 1 previous similar message
[ 7144.579400] Lustre: Evicted from MGS (at 192.168.201.104@tcp) after server handle changed from 0x9d9168dda0f6d069 to 0x9d9168dda1067485
[ 7144.584518] Lustre: Skipped 1 previous similar message
[ 7144.590973] Lustre: MGC192.168.201.104@tcp: Connection restored to 192.168.201.104@tcp (at 192.168.201.104@tcp)
[ 7144.596886] Lustre: Skipped 9 previous similar messages
[ 7168.928117] Lustre: 2363:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787379989/real 1787379989]  req@ffff92076e19df80 x1874196454286208/t0(0) o400->lustre-MDT0000-mdc-ffff92075049f000@192.168.201.104@tcp:12/10 lens 224/224 e 0 to 1 dl 1787380029 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7168.954515] Lustre: 2363:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 23 previous similar messages
[ 7175.415541] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 02:27:14 (1787380034)
[ 7177.208955] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7179.660877] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 02:27:17 (1787380037)
[ 7181.954099] LustreError: lustre-MDT0000-mdc-ffff92075049f000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7190.569166] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 02:27:29 (1787380049)
[ 7357.420445] LustreError: lustre-OST0000-osc-ffff92075049f000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7359.655973] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 02:30:18 (1787380218)
[ 7385.056144] Lustre: 2361:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787380225/real 1787380225]  req@ffff920748104700 x1874196454307328/t0(0) o4->lustre-OST0000-osc-ffff92075049f000@192.168.201.104@tcp:6/4 lens 4584/448 e 0 to 1 dl 1787380245 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7385.106162] Lustre: 2361:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 7403.984554] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 02:31:03 (1787380263)
[ 7406.103758] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7408.140532] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 02:31:06 (1787380266)
[ 7409.665246] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7411.468328] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 02:31:10 (1787380270)
[ 7443.638972] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 02:31:42 (1787380302)
[ 7482.364908] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 192.168.201.104@tcp) was lost; in progress operations using this service will fail
[ 7482.415761] Lustre: Evicted from MGS (at 192.168.201.104@tcp) after server handle changed from 0x9d9168dda1067485 to 0x9d9168dda1069cc5
[ 7482.427437] Lustre: Skipped 1 previous similar message
[ 7499.043769] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 02:32:37 (1787380357)
[ 7500.490724] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7502.371898] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 02:32:41 (1787380361)
[ 7504.068819] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7505.568619] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 02:32:44 (1787380364)
[ 7515.910791] Lustre: Unmounted lustre-client
[ 7517.824631] Lustre: Mounted lustre-client - version 2.17.57_45_gb800aec
[ 7520.282799] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7539.425548] Lustre: 2361:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787380383/real 1787380383]  req@ffff920744f16a00 x1874196454494208/t0(0) o400->MGC192.168.201.104@tcp@192.168.201.104@tcp:26/25 lens 224/224 e 0 to 1 dl 1787380399 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7539.457618] Lustre: 2361:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 7554.013809] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2