[ 1661.555917] LustreError: lustre-OST0000-osc-ffff908f107e8000: operation ost_sync to node 192.168.204.116@tcp failed: rc = -107
[ 1661.565901] LustreError: lustre-OST0000-osc-ffff908f107e8000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1661.587180] Lustre: 2395:0:(llite_lib.c:4198:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.116@tcp:/lustre/fid: [0x200000405:0x2f:0x0]// may get corrupted (rc -108)
[ 1666.711462] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 17:44:01 (1781819041)
[ 1668.623258] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1669.991365] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 17:44:04 (1781819044)
[ 1671.339165] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1672.610638] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 17:44:07 (1781819047)
[ 1675.368707] LustreError: lustre-MDT0000-mdc-ffff908f107e8000: operation mds_reint to node 192.168.204.116@tcp failed: rc = -19
[ 1695.210574] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 192.168.204.116@tcp) was lost; in progress operations using this service will fail
[ 1695.246467] Lustre: Evicted from MGS (at 192.168.204.116@tcp) after server handle changed from 0x3f3181562be262bc to 0x3f3181562be28212
[ 1789.228442] LustreError: lustre-MDT0000-mdc-ffff908f107e8000: operation ldlm_enqueue to node 192.168.204.116@tcp failed: rc = -19
[ 1807.843384] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 192.168.204.116@tcp) was lost; in progress operations using this service will fail
[ 1807.861602] Lustre: Evicted from MGS (at 192.168.204.116@tcp) after server handle changed from 0x3f3181562be28212 to 0x3f3181562be5b344
[ 1814.881716] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 17:46:29 (1781819189)
[ 1815.244707] Lustre: *** cfs_fail_loc=305, val=0***
[ 1815.247674] Lustre: Skipped 4 previous similar messages
[ 1853.919266] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 192.168.204.116@tcp) was lost; in progress operations using this service will fail
[ 1853.939054] Lustre: Evicted from MGS (at 192.168.204.116@tcp) after server handle changed from 0x3f3181562be5b344 to 0x3f3181562be5b8b5
[ 1860.566417] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1861.552096] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1866.844693] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 17:47:21 (1781819241)
[ 1874.415862] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 192.168.204.116@tcp) was lost; in progress operations using this service will fail
[ 1874.426877] Lustre: Evicted from MGS (at 192.168.204.116@tcp) after server handle changed from 0x3f3181562be5b8b5 to 0x3f3181562be5bbfd
[ 1878.793732] LustreError: lustre-MDT0000-mdc-ffff908f107e8000: operation ldlm_enqueue to node 192.168.204.116@tcp failed: rc = -107
[ 1879.535096] LustreError: lustre-MDT0000-mdc-ffff908f107e8000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 1900.086532] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 17:47:54 (1781819274)
[ 1907.437411] LustreError: lustre-OST0000-osc-ffff908f107e8000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1923.859910] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 17:48:18 (1781819298)
[ 1954.325127] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 192.168.204.116@tcp) was lost; in progress operations using this service will fail
[ 1954.351863] Lustre: Evicted from MGS (at 192.168.204.116@tcp) after server handle changed from 0x3f3181562be5bbfd to 0x3f3181562be6b747
[ 1956.394587] Lustre: 16787:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.204.116@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 1972.267236] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1973.279196] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2056.686299] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 192.168.204.116@tcp) was lost; in progress operations using this service will fail
[ 2056.707045] Lustre: Evicted from MGS (at 192.168.204.116@tcp) after server handle changed from 0x3f3181562be6b747 to 0x3f3181562bea714d
[ 2061.692075] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2062.805429] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2128.358277] Lustre: lustre-MDT0000-mdc-ffff908f107e8000: Connection to lustre-MDT0000 (at 192.168.204.116@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2128.371705] Lustre: Skipped 14 previous similar messages
[ 2144.735109] Lustre: 2392:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781819503/real 1781819503]  req@ffff908f182a3b80 x1868370992970240/t0(0) o400->MGC192.168.204.116@tcp@192.168.204.116@tcp:26/25 lens 224/224 e 0 to 1 dl 1781819519 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2144.767870] Lustre: 2392:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
[ 2144.774201] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 192.168.204.116@tcp) was lost; in progress operations using this service will fail
[ 2151.647401] Lustre: lustre-MDT0000-mdc-ffff908f107e8000: Connection restored to 192.168.204.116@tcp (at 192.168.204.116@tcp)
[ 2151.654716] Lustre: Skipped 20 previous similar messages
[ 2154.739671] Lustre: Evicted from MGS (at 192.168.204.116@tcp) after server handle changed from 0x3f3181562bea714d to 0x3f3181562bed7a4d
[ 2158.251283] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2159.922799] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2187.138965] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 17:52:41 (1781819561)
[ 2190.258809] LustreError: lustre-MDT0000-mdc-ffff908f107e8000: operation mds_reint to node 192.168.204.116@tcp failed: rc = -19
[ 2210.659581] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2232.814992] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 192.168.204.116@tcp) was lost; in progress operations using this service will fail
[ 2232.833825] LustreError: Skipped 1 previous similar message
[ 2232.856925] Lustre: Evicted from MGS (at 192.168.204.116@tcp) after server handle changed from 0x3f3181562bee5584 to 0x3f3181562bee66a9
[ 2232.866967] Lustre: Skipped 1 previous similar message
[ 2235.988377] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2242.377329] LustreError: lustre-MDT0000-mdc-ffff908f107e8000: operation mds_reint to node 192.168.204.116@tcp failed: rc = -19
[ 2242.389282] LustreError: Skipped 1 previous similar message
[ 2262.610749] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2299.584768] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2321.387346] LustreError: lustre-MDT0000-mdc-ffff908f107e8000: operation ldlm_enqueue to node 192.168.204.116@tcp failed: rc = -19
[ 2321.392076] LustreError: Skipped 3 previous similar messages
[ 2341.461122] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2387.280061] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 192.168.204.116@tcp) was lost; in progress operations using this service will fail
[ 2387.293076] LustreError: Skipped 3 previous similar messages
[ 2387.316967] Lustre: Evicted from MGS (at 192.168.204.116@tcp) after server handle changed from 0x3f3181562befc806 to 0x3f3181562bf09b2c
[ 2387.330609] Lustre: Skipped 3 previous similar messages
[ 2394.852648] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2471.553634] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 17:57:26 (1781819846)
[ 2483.589764] LustreError: lustre-OST0000-osc-ffff908f107e8000: operation ldlm_enqueue to node 192.168.204.116@tcp failed: rc = -107
[ 2483.598023] LustreError: Skipped 4 previous similar messages
[ 2508.815396] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2509.871113] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 2814.532167] LustreError: lustre-OST0000-osc-ffff908f107e8000: operation ldlm_enqueue to node 192.168.204.116@tcp failed: rc = -107
[ 2814.536030] Lustre: lustre-OST0000-osc-ffff908f107e8000: Connection to lustre-OST0000 (at 192.168.204.116@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2814.541191] Lustre: Skipped 8 previous similar messages
[ 2829.539282] Lustre: lustre-OST0000-osc-ffff908f107e8000: Connection restored to 192.168.204.116@tcp (at 192.168.204.116@tcp)
[ 2829.543733] Lustre: Skipped 16 previous similar messages
[ 2835.605096] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2836.514335] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3170.745744] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3171.850047] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3439.284232] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 18:13:33 (1781820813)
[ 3456.479160] Lustre: 51947:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781820815/real 1781820815]  req@ffff908f048e8a80 x1868371028428160/t0(0) o101->lustre-MDT0000-mdc-ffff908f107e8000@192.168.204.116@tcp:12/10 lens 576/1152 e 0 to 1 dl 1781820831 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3456.494932] Lustre: 51947:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 3456.499403] Lustre: lustre-MDT0000-mdc-ffff908f107e8000: Connection to lustre-MDT0000 (at 192.168.204.116@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3456.506383] Lustre: Skipped 1 previous similar message
[ 3456.527809] Lustre: lustre-MDT0000-mdc-ffff908f107e8000: Connection restored to 192.168.204.116@tcp (at 192.168.204.116@tcp)
[ 3456.535704] Lustre: Skipped 1 previous similar message
[ 3460.205758] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 18:13:54 (1781820834)
[ 3481.978743] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 18:14:16 (1781820856)
[ 3503.139353] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 18:14:37 (1781820877)
[ 3503.354549] Lustre: Mounted lustre-client
[ 3534.312594] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 192.168.204.116@tcp) was lost; in progress operations using this service will fail
[ 3534.318770] LustreError: Skipped 1 previous similar message
[ 3534.331810] Lustre: Evicted from MGS (at 192.168.204.116@tcp) after server handle changed from 0x3f3181562bf1a7f6 to 0x3f3181562c186d5c
[ 3534.339703] Lustre: Skipped 1 previous similar message
[ 3537.777804] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3538.683520] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3540.248622] Lustre: Unmounted lustre-client
[ 3542.913444] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 18:15:17 (1781820917)
[ 3562.975187] Lustre: 2395:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781820922/real 1781820922]  req@ffff908f0448b800 x1868371028471552/t0(0) o4->lustre-OST0000-osc-ffff908f107e8000@192.168.204.116@tcp:6/4 lens 488/448 e 0 to 1 dl 1781820938 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 3562.996366] Lustre: 2395:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
[ 3634.280875] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 18:16:48 (1781821008)
[ 3679.166817] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 18:17:33 (1781821053)
[ 3681.044830] Lustre: Unmounted lustre-client
[ 3699.724269] Lustre: Mounted lustre-client
[ 3702.919512] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 18:17:57 (1781821077)
[ 3703.071050] LustreError: 57800:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 3704.121833] LustreError: 57800:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 3704.198132] Lustre: *** cfs_fail_loc=305, val=0***
[ 3723.521323] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 18:18:18 (1781821098)
[ 3723.802079] Lustre: Mounted lustre-client
[ 3725.064194] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 3735.361109] Lustre: Unmounted lustre-client
[ 3738.523705] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 18:18:33 (1781821113)
[ 3782.943584] LustreError: lustre-MDT0000-mdc-ffff908f183a1000: operation mds_reint to node 192.168.204.116@tcp failed: rc = -19
[ 3782.953439] LustreError: Skipped 1 previous similar message
[ 3801.579182] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 192.168.204.116@tcp) was lost; in progress operations using this service will fail
[ 3801.596386] Lustre: Evicted from MGS (at 192.168.204.116@tcp) after server handle changed from 0x3f3181562c187861 to 0x3f3181562c1b3b7d
[ 3875.723532] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 18:20:50 (1781821250)
[ 3880.514710] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 3888.617678] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 192.168.204.116@tcp) was lost; in progress operations using this service will fail
[ 3888.635233] Lustre: Evicted from MGS (at 192.168.204.116@tcp) after server handle changed from 0x3f3181562c1b3b7d to 0x3f3181562c1ee289
[ 3890.037332] LustreError: lustre-MDT0000-mdc-ffff908f183a1000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 3904.560128] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 18:21:19 (1781821279)
[ 3904.790075] Lustre: Mounted lustre-client
[ 3910.125872] LustreError: lustre-OST0000-osc-ffff908f183a1000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 3921.375236] Lustre: 2394:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781821280/real 1781821280]  req@ffff908f046bd180 x1868371032760704/t0(0) o103->lustre-OST0000-osc-ffff908f11d23800@192.168.204.116@tcp:17/18 lens 328/224 e 0 to 1 dl 1781821296 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295
[ 3921.389901] Lustre: 2394:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 35 previous similar messages
[ 3922.016469] Lustre: Unmounted lustre-client
[ 3925.167658] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 18:21:39 (1781821299)
[ 3930.492332] LustreError: lustre-MDT0000-mdc-ffff908f183a1000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 3930.500352] LustreError: 62470:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 3934.381460] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 18:21:48 (1781821308)
[ 3934.471295] LustreError: 63180:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 3938.184850] LustreError: lustre-MDT0000-mdc-ffff908f183a1000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 3938.192881] LustreError: 63199:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 3938.197845] LustreError: 63180:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=1278
[ 3938.202902] LustreError: 63180:0:(import.c:719:ptlrpc_connect_import_locked()) already connecting
[ 3938.222973] LustreError: 63204:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3939.261637] LustreError: 63210:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3939.266816] LustreError: 63210:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 3941.409511] LustreError: 63232:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3941.414133] LustreError: 63232:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 3945.637709] LustreError: 63277:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3945.642556] LustreError: 63277:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 3952.354085] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 18:22:07 (1781821327)
[ 3980.056664] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3981.070986] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3988.230101] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 18:22:42 (1781821362)
[ 4015.156905] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4016.028970] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4021.535332] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 18:23:16 (1781821396)
[ 4066.824665] Lustre: lustre-OST0000-osc-ffff908f183a1000: Connection restored to 192.168.204.116@tcp (at 192.168.204.116@tcp)
[ 4066.831079] Lustre: Skipped 23 previous similar messages
[ 4072.673470] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4073.393333] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4078.084518] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 18:24:12 (1781821452)
[ 4082.152528] Lustre: lustre-OST0000-osc-ffff908f183a1000: Connection to lustre-OST0000 (at 192.168.204.116@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4082.165192] Lustre: Skipped 22 previous similar messages
[ 4102.394391] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4103.326204] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4106.468606] Lustre: Unmounted lustre-client
[ 4138.683530] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4140.451180] Lustre: Mounted lustre-client
[ 4143.781876] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 18:25:18 (1781821518)
[ 4145.196343] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 4146.251277] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 18:25:20 (1781821520)
[ 4163.429881] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 18:25:38 (1781821538)
[ 4164.147124] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 4164.971700] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 18:25:39 (1781821539)
[ 4165.742441] Lustre: *** cfs_fail_loc=805, val=0***
[ 4165.779130] Lustre: Mounted lustre-client
[ 4170.285303] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4191.711274] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 192.168.204.116@tcp) was lost; in progress operations using this service will fail
[ 4191.728401] Lustre: Evicted from MGS (at 192.168.204.116@tcp) after server handle changed from 0x3f3181562c1ef123 to 0x3f3181562c1ef552
[ 4191.740091] LustreError: 2391:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff908f104fe800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff908f18f82700) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4193.650315] Lustre: Unmounted lustre-client
[ 4198.593792] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 18:26:12 (1781821572)
[ 4217.338523] LustreError: 2391:0:(client.c:3447:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff908f048ec700 x1868371032843136/t94489280516(94489280516) o101->lustre-MDT0000-mdc-ffff908f11d21800@192.168.204.116@tcp:12/10 lens 520/664 e 0 to 0 dl 1781821652 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 4225.250405] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4226.208893] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4231.188205] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 18:26:45 (1781821605)
[ 4237.805808] LustreError: 2391:0:(osc_request.c:1024:osc_init_grant()) lustre-OST0000-osc-ffff908f11d21800: granted 8437760 but already consumed 12582912
[ 4237.813619] LustreError: lustre-OST0000-osc-ffff908f11d21800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4237.827083] Lustre: 2395:0:(llite_lib.c:4198:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.116@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 4237.952611] LustreError: 74485:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-OST0000-osc-ffff908f11d21800: namespace resource [0x280000401:0x1442:0x0].0x0 (ffff908f068fb200) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4237.963849] LustreError: 74485:0:(ldlm_resource.c:1180:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4242.574491] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 18:26:57 (1781821617)
[ 4303.839162] Lustre: 75146:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781821619/real 1781821619]  req@ffff908f046bc700 x1868371032855936/t0(0) o101->lustre-MDT0000-mdc-ffff908f11d21800@192.168.204.116@tcp:12/10 lens 576/1152 e 0 to 1 dl 1781821679 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0
[ 4303.852882] Lustre: 75146:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 4307.900282] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 18:28:02 (1781821682)
[ 4373.395429] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 18:29:07 (1781821747)
[ 4395.846438] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 18:29:30 (1781821770)
[ 4461.472432] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 18:30:35 (1781821835)
[ 4526.778637] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 18:31:41 (1781821901)
[ 4550.154240] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 18:32:04 (1781821924)
[ 4615.169635] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 18:33:09 (1781821989)
[ 4636.420369] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 18:33:31 (1781822011)
[ 4657.871308] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 18:33:52 (1781822032)
[ 4679.784644] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 18:34:14 (1781822054)
[ 4680.052820] Lustre: Unmounted lustre-client
[ 4705.167445] LustreError: lustre-MDT0001-mdc-ffff908f07d98000: operation mds_connect to node 192.168.204.116@tcp failed: rc = -16
[ 4705.174774] LustreError: Skipped 2 previous similar messages
[ 4705.184871] Lustre: Mounted lustre-client
[ 4721.742467] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 18:34:56 (1781822096)
[ 4735.367130] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 18:35:09 (1781822109)
[ 4741.095068] Lustre: lustre-MDT0000-mdc-ffff908f07d98000: Connection to lustre-MDT0000 (at 192.168.204.116@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4741.104236] Lustre: Skipped 10 previous similar messages
[ 4746.225611] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 192.168.204.116@tcp) was lost; in progress operations using this service will fail
[ 4746.243562] LustreError: Skipped 1 previous similar message
[ 4746.254613] Lustre: Evicted from MGS (at 192.168.204.116@tcp) after server handle changed from 0x3f3181562c1f1af2 to 0x3f3181562c1f21eb
[ 4746.261055] Lustre: Skipped 1 previous similar message
[ 4746.263962] Lustre: MGC192.168.204.116@tcp: Connection restored to 192.168.204.116@tcp (at 192.168.204.116@tcp)
[ 4746.267690] Lustre: Skipped 12 previous similar messages
[ 4753.198483] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 18:35:27 (1781822127)
[ 4780.329406] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 18:35:54 (1781822154)
[ 4780.761198] Lustre: *** cfs_fail_loc=51b, val=3***
[ 4788.795791] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 18:36:03 (1781822163)
[ 4789.080424] Lustre: *** cfs_fail_loc=51b, val=4***
[ 4797.407769] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 18:36:11 (1781822171)
[ 4797.595869] Lustre: *** cfs_fail_loc=50f, val=3***
[ 4802.856716] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 18:36:17 (1781822177)
[ 4803.030786] Lustre: *** cfs_fail_loc=50f, val=4***
[ 4808.671230] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 18:36:23 (1781822183)
[ 4808.831544] Lustre: *** cfs_fail_loc=510, val=3***
[ 4813.840640] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 18:36:28 (1781822188)
[ 4814.059718] Lustre: *** cfs_fail_loc=51b, val=3***
[ 4821.190428] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 18:36:35 (1781822195)
[ 4884.233336] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 18:37:38 (1781822258)
[ 4884.434521] LustreError: 89273:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 4886.919785] LustreError: lustre-MDT0000-mdc-ffff908f07d98000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4886.932348] LustreError: 89287:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff908f07d98000: inode [0x20000afe2:0x6:0x0] mdc close failed: rc = -108
[ 4886.944603] LustreError: 89287:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff908f07d98000: namespace resource [0x20000afe2:0xc:0x0].0xc (ffff908f107e5300) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4888.503262] LustreError: 89273:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 4890.547334] LustreError: 89304:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 4893.036065] LustreError: lustre-MDT0000-mdc-ffff908f07d98000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4893.052248] LustreError: 89318:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff908f07d98000: namespace resource [0x20000afe2:0xc:0x0].0xc (ffff908f107e5c00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4893.061032] LustreError: 89318:0:(ldlm_resource.c:1180:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4894.615377] LustreError: 89304:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 4900.990488] LustreError: lustre-MDT0000-mdc-ffff908f07d98000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4901.170426] LustreError: 89359:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 4903.570569] LustreError: lustre-MDT0000-mdc-ffff908f07d98000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4903.607264] LustreError: 89379:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4903.614990] LustreError: 89379:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 20 previous similar messages
[ 4904.661237] LustreError: 89385:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4904.666540] LustreError: 89385:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4905.231232] LustreError: 89359:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 4905.238670] LustreError: 89359:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff908f07d98000: inode [0x20000afe2:0xc:0x0] mdc close failed: rc = -108
[ 4905.246706] LustreError: 89359:0:(file.c:251:ll_close_inode_openhandle()) Skipped 3 previous similar messages
[ 4906.850355] LustreError: 89407:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4906.856198] LustreError: 89407:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4910.191544] LustreError: 89434:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 4912.751313] LustreError: lustre-MDT0000-mdc-ffff908f07d98000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4912.763937] LustreError: 89448:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff908f07d98000: namespace resource [0x20000afe2:0xc:0x0].0xc (ffff908f18351000) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4912.780470] LustreError: 89448:0:(ldlm_resource.c:1180:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4914.256339] LustreError: 89434:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 4920.779972] LustreError: lustre-MDT0000-mdc-ffff908f07d98000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4921.071659] LustreError: 89489:0:(ldlm_flock.c:858:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 4923.626707] LustreError: lustre-MDT0000-mdc-ffff908f07d98000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4923.642106] LustreError: 89504:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff908f07d98000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff908f107e5500) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4923.659682] LustreError: 89504:0:(ldlm_resource.c:1180:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4925.135331] LustreError: 89489:0:(ldlm_flock.c:858:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 4927.841071] LustreError: lustre-MDT0000-mdc-ffff908f07d98000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4927.881324] LustreError: 89537:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4927.890144] LustreError: 89537:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 4929.288455] LustreError: 89517:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff908f07d98000: inode [0x20000afe2:0xc:0x0] mdc close failed: rc = -108
[ 4936.538161] LustreError: 89588:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 4936.544731] LustreError: 89588:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 4939.492179] LustreError: lustre-MDT0000-mdc-ffff908f07d98000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4939.540166] LustreError: 89610:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4939.545752] LustreError: 89610:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 4940.608230] LustreError: 89588:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 4940.614715] LustreError: 89588:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 4940.619571] LustreError: 89588:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff908f07d98000: inode [0x20000afe2:0xc:0x0] mdc close failed: rc = -108
[ 4947.771430] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 18:38:42 (1781822322)
[ 5008.351181] Lustre: 90307:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781822323/real 1781822323]  req@ffff908f1820f480 x1868371033069952/t0(0) o101->lustre-MDT0000-mdc-ffff908f07d98000@192.168.204.116@tcp:12/10 lens 576/1152 e 0 to 1 dl 1781822383 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'stat.0' uid:0 gid:0 projid:0
[ 5008.372164] Lustre: 90307:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
[ 5009.455904] LustreError: 81705:0:(ldlm_lockd.c:2883:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5009.465924] LustreError: 81705:0:(ldlm_lockd.c:2883:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5013.535221] LustreError: 81705:0:(ldlm_lockd.c:2883:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5013.541734] LustreError: 81705:0:(ldlm_lockd.c:2883:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5017.518328] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 18:39:52 (1781822392)
[ 5083.459246] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 18:40:57 (1781822457)
[ 5150.295207] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 18:42:04 (1781822524)
[ 5179.720565] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 18:42:34 (1781822554)
[ 5179.951688] Lustre: Mounted lustre-client
[ 5302.423227] Lustre: Unmounted lustre-client
[ 5316.821649] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 18:44:51 (1781822691)
[ 5317.289624] LustreError: 94006:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5319.149105] LustreError: lustre-OST0000-osc-ffff908f07d98000: operation ldlm_enqueue to node 192.168.204.116@tcp failed: rc = -107
[ 5319.156721] LustreError: Skipped 9 previous similar messages
[ 5319.163243] LustreError: lustre-OST0000-osc-ffff908f07d98000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5319.171366] LustreError: 94088:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-OST0000-osc-ffff908f07d98000: namespace resource [0x280000401:0x146d:0x0].0x0 (ffff908f107e5300) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5319.177967] LustreError: 94088:0:(ldlm_resource.c:1180:ldlm_resource_complain()) Skipped 2 previous similar messages
[ 5319.375140] LustreError: 94006:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5319.378537] Lustre: 2394:0:(llite_lib.c:4198:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.116@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5325.318425] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 18:44:58 (1781822698)
[ 5350.367221] Lustre: lustre-MDT0000-mdc-ffff908f07d98000: Connection to lustre-MDT0000 (at 192.168.204.116@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5350.381763] Lustre: Skipped 14 previous similar messages
[ 5350.391975] Lustre: lustre-MDT0000-mdc-ffff908f07d98000: Connection restored to 192.168.204.116@tcp (at 192.168.204.116@tcp)
[ 5350.396166] Lustre: Skipped 13 previous similar messages
[ 5371.887032] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 18:45:46 (1781822746)
[ 5373.588623] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 5375.479683] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 18:45:49 (1781822749)
[ 5405.247450] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 5406.721408] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 18:46:20 (1781822780)
[ 5431.449799] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 18:46:45 (1781822805)
[ 5433.192438] Lustre: Unmounted lustre-client
[ 5549.991528] Lustre: Mounted lustre-client
[ 5556.128640] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 18:48:50 (1781822930)
[ 5570.551312] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 192.168.204.116@tcp) was lost; in progress operations using this service will fail
[ 5570.582766] Lustre: Evicted from MGS (at 192.168.204.116@tcp) after server handle changed from 0x3f3181562c1f3e5b to 0x3f3181562c1f4099
[ 5583.830650] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 18:49:17 (1781822957)
[ 5617.992542] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 18:49:52 (1781822992)
[ 5634.398890] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5662.687191] Lustre: 2392:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781823022/real 1781823022]  req@ffff908f048eea00 x1868371033193856/t0(0) o400->MGC192.168.204.116@tcp@192.168.204.116@tcp:26/25 lens 224/224 e 0 to 1 dl 1781823038 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5662.740301] Lustre: 2392:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
[ 5683.577086] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5685.507533] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5695.505989] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 18:51:09 (1781823069)
[ 5697.641162] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 5699.417565] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 18:51:13 (1781823073)
[ 5725.667545] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 18:51:39 (1781823099)
[ 5757.775612] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 18:52:11 (1781823131)
[ 5815.531985] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5817.545518] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5919.366370] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5921.503922] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5956.627783] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5958.117659] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6043.336360] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 18:56:57 (1781823417)
[ 6058.971565] LustreError: lustre-OST0000-osc-ffff908f082eb800: operation ost_setattr to node 192.168.204.116@tcp failed: rc = -19
[ 6058.974729] LustreError: Skipped 13 previous similar messages
[ 6058.980491] Lustre: lustre-OST0000-osc-ffff908f082eb800: Connection to lustre-OST0000 (at 192.168.204.116@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6059.000490] Lustre: Skipped 9 previous similar messages
[ 6113.530388] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 6117.387476] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6265.056587] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 19:00:38 (1781823638)
[ 6437.352301] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 192.168.204.116@tcp) was lost; in progress operations using this service will fail
[ 6437.357985] LustreError: Skipped 5 previous similar messages
[ 6437.394107] Lustre: Evicted from MGS (at 192.168.204.116@tcp) after server handle changed from 0x3f3181562c1f6281 to 0x3f3181562c2f067a
[ 6437.397584] Lustre: Skipped 5 previous similar messages
[ 6437.400967] Lustre: MGC192.168.204.116@tcp: Connection restored to 192.168.204.116@tcp (at 192.168.204.116@tcp)
[ 6437.404288] Lustre: Skipped 14 previous similar messages
[ 6473.091894] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 19:04:06 (1781823846)
[ 6475.094469] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 6477.166512] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 19:04:10 (1781823850)
[ 6479.478482] LustreError: lustre-MDT0000-mdc-ffff908f082eb800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6490.364102] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 19:04:24 (1781823864)
[ 6658.036316] LustreError: lustre-OST0000-osc-ffff908f082eb800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6659.767895] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 19:07:13 (1781824033)
[ 6686.687841] Lustre: 2394:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781824041/real 1781824041]  req@ffff908f06490e00 x1868371050889472/t0(0) o4->lustre-OST0000-osc-ffff908f082eb800@192.168.204.116@tcp:6/4 lens 4584/448 e 0 to 1 dl 1781824061 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 6686.711131] Lustre: 2394:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 6686.719295] Lustre: lustre-OST0000-osc-ffff908f082eb800: Connection to lustre-OST0000 (at 192.168.204.116@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6686.730394] Lustre: Skipped 3 previous similar messages
[ 6703.348652] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 19:07:57 (1781824077)
[ 6747.129858] LustreError: lustre-MDT0001-mdc-ffff908f082eb800: This client was evicted by lustre-MDT0001; in progress operations using this service will fail.
[ 6752.238155] LustreError: lustre-MDT0000-mdc-ffff908f082eb800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6752.246719] LustreError: 111911:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff908f082eb800: inode [0x2000105d1:0x6:0x0] mdc close failed: rc = -5
[ 6760.781639] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 19:08:54 (1781824134)
[ 6765.582774] LustreError: lustre-MDT0000-mdc-ffff908f082eb800: operation mds_statfs to node 192.168.204.116@tcp failed: rc = -107
[ 6765.592645] LustreError: Skipped 159 previous similar messages
[ 6765.610116] LustreError: 113040:0:(lmv_obd.c:1466:lmv_statfs()) lustre-MDT0000-mdc-ffff908f082eb800: can't stat MDS #0: rc = -107
[ 6790.528612] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 19:09:24 (1781824164)
[ 6824.024250] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 19:09:58 (1781824198)
[ 6881.062746] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 19:10:54 (1781824254)
[ 6939.293347] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 19:11:53 (1781824313)
[ 6957.278625] LustreError: lustre-MDT0000-mdc-ffff908f082eb800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6968.152510] Lustre: Mounted lustre-client
[ 6970.447088] Lustre: Unmounted lustre-client
[ 6989.348961] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 19:12:43 (1781824363)
[ 7000.843601] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7042.559505] Lustre: DEBUG MARKER: == recovery-small test 156: tot_granted miscount after client eviction ========================================================== 19:13:36 (1781824416)
[ 7052.505895] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-OST0000
[ 7079.478277] LustreError: 2391:0:(client.c:3532:ptlrpc_replay_req()) cfs_fail_timeout id 536 sleeping for 45000ms
[ 7124.519176] LustreError: 2391:0:(client.c:3532:ptlrpc_replay_req()) cfs_fail_timeout id 536 awake
[ 7124.559925] LustreError: lustre-OST0000-osc-ffff908f1166f000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7130.790407] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 7132.392333] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 7142.695245] Lustre: DEBUG MARKER: == recovery-small test 157: eviction during mmaped i/o === 19:15:16 (1781824516)
[ 7143.264198] LustreError: 120202:0:(vvp_io.c:1473:vvp_io_fault_start()) cfs_fail_timeout id 1432 sleeping for 3000ms
[ 7146.296791] LustreError: 120202:0:(vvp_io.c:1473:vvp_io_fault_start()) cfs_fail_timeout id 1432 awake
[ 7146.317674] LustreError: lustre-OST0000-osc-ffff908f1166f000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7146.346113] Lustre: lustre-OST0000-osc-ffff908f1166f000: Connection restored to 192.168.204.116@tcp (at 192.168.204.116@tcp)
[ 7146.358329] Lustre: Skipped 16 previous similar messages
[ 7152.706536] Lustre: DEBUG MARKER: == recovery-small test 158a: connect without access right ========================================================== 19:15:26 (1781824526)
[ 7154.681248] Lustre: Unmounted lustre-client
[ 7154.688211] Lustre: Skipped 2 previous similar messages
[ 7155.261333] Lustre: Mounted lustre-client
[ 7155.270529] Lustre: Skipped 2 previous similar messages
[ 7180.153793] Lustre: lustre-MDT0001-mdc-ffff908f11d21000: connection denied by lustre-MDT0001_UUID: rc = -13
[ 7180.171393] LustreError: lustre-MDT0001-mdc-ffff908f11d21000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail.
[ 7199.733906] LustreError: lustre-MDT0001-mdc-ffff908f11d21000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail.
[ 7212.460628] Lustre: DEBUG MARKER: == recovery-small test 160: MDT destroys are blocked by grouplocks ========================================================== 19:16:26 (1781824586)
[ 7215.887647] LustreError: lustre-MDT0000-mdc-ffff908f02d98000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7261.686919] Lustre: DEBUG MARKER: == recovery-small test 161: evict osp by ping evictor ==== 19:17:16 (1781824636)
[ 7280.172331] Lustre: DEBUG MARKER: == recovery-small test 162: File attributes should be persisted after MDS failover ========================================================== 19:17:34 (1781824654)
[ 7295.496599] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 192.168.204.116@tcp) was lost; in progress operations using this service will fail
[ 7295.520420] LustreError: Skipped 6 previous similar messages
[ 7295.526855] LustreError: 2391:0:(client.c:3447:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff908f0609b100 x1868371051265792/t163208757317(163208757317) o101->lustre-MDT0000-mdc-ffff908f02d98000@192.168.204.116@tcp:12/10 lens 576/608 e 0 to 0 dl 1781824725 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'chattr.0' uid:0 gid:0 projid:0
[ 7295.586244] Lustre: Evicted from MGS (at 192.168.204.116@tcp) after server handle changed from 0x3f3181562c2f6348 to 0x3f3181562c2f6d89
[ 7295.598862] Lustre: Skipped 7 previous similar messages
[ 7311.133719] Lustre: DEBUG MARKER: == recovery-small test 163: changelog check for fail write and processing records ========================================================== 19:18:04 (1781824684)
[ 7338.312847] Lustre: DEBUG MARKER: == recovery-small test 170: Reconnect after REPLAY_LOCKS hangs (LU-18154) ========================================================== 19:18:32 (1781824712)
[ 7345.510449] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-OST0000
[ 7349.238389] Lustre: lustre-OST0000-osc-ffff908f02d98000: Connection to lustre-OST0000 (at 192.168.204.116@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7349.248333] Lustre: Skipped 13 previous similar messages
[ 7381.236390] Lustre: *** cfs_fail_loc=537, val=0***
[ 7381.241875] LustreError: 2391:0:(import.c:719:ptlrpc_connect_import_locked()) already connecting
[ 7382.000198] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount REPLAY_LOCKS osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 8960.036318] Lustre: DEBUG MARKER: rpc test_170: @@@@@@ FAIL: can't put import for osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid into REPLAY_LOCKS state after 1475 sec, have IDLE