[ 1940.852266] LustreError: lustre-OST0000-osc-ffff952b48fc3800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1940.893843] Lustre: 2349:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.147@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1948.999816] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 14:31:30 (1787855490)
[ 1951.206694] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1953.236801] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 14:31:34 (1787855494)
[ 1955.072346] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1957.340648] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 14:31:38 (1787855498)
[ 1961.402048] LustreError: lustre-MDT0000-mdc-ffff952b48fc3800: operation ldlm_enqueue to node 192.168.204.147@tcp failed: rc = -19
[ 1961.414748] LustreError: Skipped 1 previous similar message
[ 1978.336296] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 192.168.204.147@tcp) was lost; in progress operations using this service will fail
[ 1988.533224] Lustre: Evicted from MGS (at 192.168.204.147@tcp) after server handle changed from 0xce1e7d5b568e4b1f to 0xce1e7d5b568e602d
[ 1989.619508] Lustre: 15944:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.147@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2099.058242] LustreError: lustre-MDT0000-mdc-ffff952b48fc3800: operation mds_close to node 192.168.204.147@tcp failed: rc = -19
[ 2099.077790] LustreError: Skipped 1 previous similar message
[ 2121.248665] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 192.168.204.147@tcp) was lost; in progress operations using this service will fail
[ 2131.563606] Lustre: Evicted from MGS (at 192.168.204.147@tcp) after server handle changed from 0xce1e7d5b568e602d to 0xce1e7d5b56901996
[ 2158.781155] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 14:35:00 (1787855700)
[ 2159.749230] Lustre: *** cfs_fail_loc=305, val=0***
[ 2159.754115] Lustre: Skipped 2 previous similar messages
[ 2203.617636] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 192.168.204.147@tcp) was lost; in progress operations using this service will fail
[ 2203.671062] Lustre: Evicted from MGS (at 192.168.204.147@tcp) after server handle changed from 0xce1e7d5b56901996 to 0xce1e7d5b56901bf7
[ 2203.692517] Lustre: MGC192.168.204.147@tcp: Connection restored to 192.168.204.147@tcp (at 192.168.204.147@tcp)
[ 2203.706816] Lustre: Skipped 13 previous similar messages
[ 2204.921661] Lustre: 15944:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.147@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2213.883946] Lustre: lustre-MDT0000-mdc-ffff952b48fc3800: Connection to lustre-MDT0000 (at 192.168.204.147@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2213.906204] Lustre: Skipped 11 previous similar messages
[ 2232.120681] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2234.651986] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2245.117932] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 14:36:26 (1787855786)
[ 2259.983216] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 192.168.204.147@tcp) was lost; in progress operations using this service will fail
[ 2260.005962] Lustre: Evicted from MGS (at 192.168.204.147@tcp) after server handle changed from 0xce1e7d5b56901bf7 to 0xce1e7d5b56901faf
[ 2261.561276] LustreError: lustre-MDT0000-mdc-ffff952b48fc3800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2287.887579] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 14:37:08 (1787855828)
[ 2302.292338] LustreError: lustre-OST0000-osc-ffff952b48fc3800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2322.826737] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 14:37:44 (1787855864)
[ 2337.751187] LustreError: lustre-MDT0000-mdc-ffff952b48fc3800: operation mds_reint to node 192.168.204.147@tcp failed: rc = -19
[ 2357.215783] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 192.168.204.147@tcp) was lost; in progress operations using this service will fail
[ 2367.419396] Lustre: Evicted from MGS (at 192.168.204.147@tcp) after server handle changed from 0xce1e7d5b56901faf to 0xce1e7d5b56906825
[ 2392.520151] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2395.544344] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2480.099854] Lustre: 2347:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787856006/real 1787856006]  req@ffff952b50267840 x1874700370558336/t0(0) o400->MGC192.168.204.147@tcp@192.168.204.147@tcp:26/25 lens 224/224 e 0 to 1 dl 1787856022 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2480.136779] Lustre: 2347:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 15 previous similar messages
[ 2480.164354] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 192.168.204.147@tcp) was lost; in progress operations using this service will fail
[ 2490.412627] Lustre: Evicted from MGS (at 192.168.204.147@tcp) after server handle changed from 0xce1e7d5b56906825 to 0xce1e7d5b5691fb0e
[ 2510.517735] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2513.262495] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2579.543281] LustreError: lustre-MDT0000-mdc-ffff952b48fc3800: operation mds_reint to node 192.168.204.147@tcp failed: rc = -19
[ 2579.561214] LustreError: Skipped 1 previous similar message
[ 2596.319583] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 192.168.204.147@tcp) was lost; in progress operations using this service will fail
[ 2606.506666] Lustre: Evicted from MGS (at 192.168.204.147@tcp) after server handle changed from 0xce1e7d5b5691fb0e to 0xce1e7d5b56939ec1
[ 2608.497250] Lustre: 15944:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.147@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2629.503027] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2631.982587] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2662.242973] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 14:43:23 (1787856203)
[ 2687.455241] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 192.168.204.147@tcp) was lost; in progress operations using this service will fail
[ 2687.484633] Lustre: Evicted from MGS (at 192.168.204.147@tcp) after server handle changed from 0xce1e7d5b56939ec1 to 0xce1e7d5b56947a76
[ 2689.147530] Lustre: 15944:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.147@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2706.418953] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2739.185494] Lustre: 15944:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.147@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2755.573351] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2800.425165] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2814.103720] Lustre: lustre-MDT0000-mdc-ffff952b48fc3800: Connection to lustre-MDT0000 (at 192.168.204.147@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2814.123682] Lustre: Skipped 8 previous similar messages
[ 2834.911896] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 192.168.204.147@tcp) was lost; in progress operations using this service will fail
[ 2834.925206] LustreError: Skipped 2 previous similar messages
[ 2834.937663] Lustre: Evicted from MGS (at 192.168.204.147@tcp) after server handle changed from 0xce1e7d5b5694dcca to 0xce1e7d5b56951d7c
[ 2834.950687] Lustre: Skipped 2 previous similar messages
[ 2834.964414] Lustre: MGC192.168.204.147@tcp: Connection restored to 192.168.204.147@tcp (at 192.168.204.147@tcp)
[ 2834.973210] Lustre: Skipped 16 previous similar messages
[ 2836.152906] Lustre: 15944:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.147@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2853.767569] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2878.318738] LustreError: lustre-MDT0000-mdc-ffff952b48fc3800: operation ldlm_enqueue to node 192.168.204.147@tcp failed: rc = -19
[ 2878.337761] LustreError: Skipped 8 previous similar messages
[ 2906.161113] Lustre: 15944:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.147@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2917.042137] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2977.765986] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 3037.617472] Lustre: 15944:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.147@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 3074.537372] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 14:50:16 (1787856616)
[ 3134.630813] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3136.116446] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3422.556062] LustreError: lustre-OST0000-osc-ffff952b48fc3800: operation ldlm_enqueue to node 192.168.204.147@tcp failed: rc = -107
[ 3422.574347] LustreError: Skipped 6 previous similar messages
[ 3422.581189] Lustre: lustre-OST0000-osc-ffff952b48fc3800: Connection to lustre-OST0000 (at 192.168.204.147@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3422.594203] Lustre: Skipped 4 previous similar messages
[ 3445.356808] Lustre: lustre-OST0000-osc-ffff952b48fc3800: Connection restored to 192.168.204.147@tcp (at 192.168.204.147@tcp)
[ 3445.372162] Lustre: Skipped 8 previous similar messages
[ 3463.475803] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3465.584707] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3796.693035] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3799.426517] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4049.927253] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 15:06:31 (1787857591)
[ 4067.813887] Lustre: 50574:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787857594/real 1787857594]  req@ffff952b47b0fc00 x1874700392097664/t0(0) o101->lustre-MDT0000-mdc-ffff952b48fc3800@192.168.204.147@tcp:12/10 lens 576/1152 e 0 to 1 dl 1787857610 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 4067.851965] Lustre: 50574:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
[ 4067.872324] Lustre: lustre-MDT0000-mdc-ffff952b48fc3800: Connection to lustre-MDT0000 (at 192.168.204.147@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4067.888689] Lustre: Skipped 1 previous similar message
[ 4067.951965] Lustre: lustre-MDT0000-mdc-ffff952b48fc3800: Connection restored to 192.168.204.147@tcp (at 192.168.204.147@tcp)
[ 4067.958726] Lustre: Skipped 1 previous similar message
[ 4076.471319] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 15:06:58 (1787857618)
[ 4100.755947] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 15:07:22 (1787857642)
[ 4128.132453] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 15:07:49 (1787857669)
[ 4129.054549] Lustre: Mounted lustre-client - version 2.17.57_83_g3de4944
[ 4160.991183] Lustre: 2347:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787857687/real 1787857687]  req@ffff952b47b0d680 x1874700392116480/t0(0) o400->lustre-MDT0000-mdc-ffff952b48fc3800@192.168.204.147@tcp:12/10 lens 224/224 e 0 to 1 dl 1787857703 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4161.052213] Lustre: 2347:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 4161.061133] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 192.168.204.147@tcp) was lost; in progress operations using this service will fail
[ 4161.081046] LustreError: Skipped 3 previous similar messages
[ 4170.225614] Lustre: Evicted from MGS (at 192.168.204.147@tcp) after server handle changed from 0xce1e7d5b5696a88c to 0xce1e7d5b56ab3cd0
[ 4170.238143] Lustre: Skipped 3 previous similar messages
[ 4192.828722] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4195.331883] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4197.905419] Lustre: Unmounted lustre-client
[ 4205.229412] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 15:09:06 (1787857746)
[ 4311.519342] Lustre: 2349:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787857838/real 1787857838]  req@ffff952b504e3480 x1874700392152832/t0(0) o4->lustre-OST0000-osc-ffff952b48fc3800@192.168.204.147@tcp:6/4 lens 488/448 e 0 to 1 dl 1787857854 ref 2 fl Rpc:XQr/602/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4311.576513] Lustre: 2349:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 59 previous similar messages
[ 4502.022140] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 15:14:03 (1787858043)
[ 4554.030603] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 15:14:54 (1787858094)
[ 4558.327030] Lustre: Unmounted lustre-client
[ 4582.448622] Lustre: Mounted lustre-client - version 2.17.57_83_g3de4944
[ 4589.291666] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 15:15:30 (1787858130)
[ 4589.595413] LustreError: 56502:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4590.647095] LustreError: 56502:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4590.802962] Lustre: *** cfs_fail_loc=305, val=0***
[ 4615.673220] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 15:15:56 (1787858156)
[ 4616.292631] Lustre: Mounted lustre-client - version 2.17.57_83_g3de4944
[ 4619.765973] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4630.125069] Lustre: Unmounted lustre-client
[ 4638.360560] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 15:16:19 (1787858179)
[ 4736.104465] LustreError: lustre-MDT0000-mdc-ffff952b47bae000: operation mds_reint to node 192.168.204.147@tcp failed: rc = -19
[ 4736.110679] LustreError: Skipped 1 previous similar message
[ 4736.118345] Lustre: lustre-MDT0000-mdc-ffff952b47bae000: Connection to lustre-MDT0000 (at 192.168.204.147@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4736.138348] Lustre: Skipped 23 previous similar messages
[ 4755.424330] Lustre: 2347:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787858282/real 1787858282]  req@ffff952b60480f00 x1874700394825984/t0(0) o400->MGC192.168.204.147@tcp@192.168.204.147@tcp:26/25 lens 224/224 e 0 to 1 dl 1787858298 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4755.473953] Lustre: 2347:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 76 previous similar messages
[ 4755.489822] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 192.168.204.147@tcp) was lost; in progress operations using this service will fail
[ 4765.669943] Lustre: Evicted from MGS (at 192.168.204.147@tcp) after server handle changed from 0xce1e7d5b56ab589f to 0xce1e7d5b56ae0a0a
[ 4765.688601] Lustre: MGC192.168.204.147@tcp: Connection restored to 192.168.204.147@tcp (at 192.168.204.147@tcp)
[ 4765.699573] Lustre: Skipped 24 previous similar messages
[ 4934.494531] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 15:21:16 (1787858476)
[ 4939.626445] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4954.090196] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 192.168.204.147@tcp) was lost; in progress operations using this service will fail
[ 4954.121406] Lustre: Evicted from MGS (at 192.168.204.147@tcp) after server handle changed from 0xce1e7d5b56ae0a0a to 0xce1e7d5b56b2d9d9
[ 4954.185631] LustreError: lustre-MDT0000-mdc-ffff952b47bae000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4968.814151] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 15:21:49 (1787858509)
[ 4969.288919] Lustre: Mounted lustre-client - version 2.17.57_83_g3de4944
[ 4976.676562] LustreError: lustre-OST0000-osc-ffff952b47bae000: operation ldlm_enqueue to node 192.168.204.147@tcp failed: rc = -107
[ 4976.693937] LustreError: lustre-OST0000-osc-ffff952b47bae000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4990.324560] Lustre: Unmounted lustre-client
[ 4999.056438] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 15:22:20 (1787858540)
[ 5005.804990] LustreError: lustre-MDT0000-mdc-ffff952b47bae000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5005.833099] LustreError: 60897:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 5014.482427] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 15:22:35 (1787858555)
[ 5014.899620] LustreError: 61602:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 5020.128893] LustreError: 61602:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=0
[ 5020.137586] LustreError: 61602:0:(import.c:711:ptlrpc_connect_import_locked()) already connected
[ 5021.701418] LustreError: lustre-MDT0000-mdc-ffff952b47bae000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5021.712851] LustreError: 61619:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 5021.784475] LustreError: 61624:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5022.895272] LustreError: 61630:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5022.912455] LustreError: 61630:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5025.224726] LustreError: 61652:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5025.232962] LustreError: 61652:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5029.944921] LustreError: 61697:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5029.971947] LustreError: 61697:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 5039.587834] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 15:23:01 (1787858581)
[ 5078.870894] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5080.561910] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5091.051445] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 15:23:52 (1787858632)
[ 5128.103116] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5129.702513] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5139.847781] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 15:24:41 (1787858681)
[ 5205.894193] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5207.471453] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5216.678323] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 15:25:57 (1787858757)
[ 5255.927657] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5257.513167] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5263.169638] Lustre: Unmounted lustre-client
[ 5316.884805] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5319.596920] Lustre: Mounted lustre-client - version 2.17.57_83_g3de4944
[ 5326.549910] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 15:27:48 (1787858868)
[ 5329.072810] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5330.922605] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 15:27:52 (1787858872)
[ 5360.687625] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 15:28:22 (1787858902)
[ 5362.141427] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5363.664738] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 15:28:25 (1787858905)
[ 5364.987661] Lustre: *** cfs_fail_loc=805, val=0***
[ 5365.038631] Lustre: Mounted lustre-client - version 2.17.57_83_g3de4944
[ 5369.745838] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5390.837088] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 192.168.204.147@tcp) was lost; in progress operations using this service will fail
[ 5390.861436] Lustre: Evicted from MGS (at 192.168.204.147@tcp) after server handle changed from 0xce1e7d5b56b2e40c to 0xce1e7d5b56b2e627
[ 5390.870940] Lustre: MGC192.168.204.147@tcp: Connection restored to 192.168.204.147@tcp (at 192.168.204.147@tcp)
[ 5390.881349] Lustre: Skipped 13 previous similar messages
[ 5391.839509] Lustre: 2348:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787858918/real 1787858918]  req@ffff952b52072940 x1874700396887296/t0(0) o400->lustre-MDT0000-mdc-ffff952b44f72800@192.168.204.147@tcp:12/10 lens 224/224 e 0 to 1 dl 1787858934 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5391.863866] Lustre: 2348:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5391.869205] Lustre: lustre-MDT0000-mdc-ffff952b44f72800: Connection to lustre-MDT0000 (at 192.168.204.147@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5391.880624] Lustre: Skipped 13 previous similar messages
[ 5392.758293] Lustre: 68263:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.147@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 5409.321450] Lustre: Unmounted lustre-client
[ 5417.166977] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 15:29:18 (1787858958)
[ 5449.846073] Lustre: 68263:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.147@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 5459.490331] LustreError: 2346:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff952b47e0c000 x1874700396899200/t94489280516(94489280516) o101->lustre-MDT0000-mdc-ffff952b47105800@192.168.204.147@tcp:12/10 lens 664/608 e 0 to 0 dl 1787859018 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5466.515887] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5468.462508] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5478.012745] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 15:30:19 (1787859019)
[ 5479.688725] LustreError: lustre-OST0000-osc-ffff952b47105800: operation ost_write to node 192.168.204.147@tcp failed: rc = -107
[ 5479.703534] LustreError: Skipped 2 previous similar messages
[ 5479.739763] LustreError: lustre-OST0000-osc-ffff952b47105800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5479.766386] Lustre: 2348:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.147@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5479.816289] LustreError: 72804:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff952b47105800: namespace resource [0x240000400:0x3d62:0x0].0x0 (ffff952b44868e00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5488.639856] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 15:30:30 (1787859030)
[ 5490.393239] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5492.732626] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 15:30:33 (1787859033)
[ 5494.227944] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5495.975607] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 15:30:37 (1787859037)
[ 5497.671690] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5499.403572] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 15:30:40 (1787859040)
[ 5500.864174] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5502.660229] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 15:30:44 (1787859044)
[ 5504.200930] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5505.745830] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 15:30:47 (1787859047)
[ 5507.329375] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5508.942447] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 15:30:50 (1787859050)
[ 5510.375737] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5511.764277] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 15:30:53 (1787859053)
[ 5513.137840] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5514.653580] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 15:30:56 (1787859056)
[ 5516.051737] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5517.941614] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 15:30:59 (1787859059)
[ 5519.321436] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5520.939942] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 15:31:02 (1787859062)
[ 5522.583149] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5524.414567] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 15:31:05 (1787859065)
[ 5526.980432] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5529.017733] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 15:31:10 (1787859070)
[ 5563.080391] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 15:31:44 (1787859104)
[ 5594.870190] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 15:32:15 (1787859135)
[ 5595.385353] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5605.815182] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 15:32:27 (1787859147)
[ 5606.323980] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5617.916599] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 15:32:39 (1787859159)
[ 5618.360512] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5627.784583] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 15:32:48 (1787859168)
[ 5628.092916] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5637.590257] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 15:32:58 (1787859178)
[ 5638.176913] Lustre: *** cfs_fail_loc=510, val=3***
[ 5646.960619] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 15:33:08 (1787859188)
[ 5647.425916] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5658.235816] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 15:33:19 (1787859199)
[ 5658.663150] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5723.899413] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 15:34:25 (1787859265)
[ 5724.262307] LustreError: 83010:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5727.410889] LustreError: lustre-MDT0000-mdc-ffff952b47105800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5727.455081] LustreError: 83025:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff952b47105800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff952b44bb9a00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5728.344056] LustreError: 83010:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5730.522136] LustreError: 83040:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5733.464772] LustreError: lustre-MDT0000-mdc-ffff952b47105800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5733.502168] LustreError: 83054:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff952b47105800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff952b50821800) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5733.552814] LustreError: 83054:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5734.527597] LustreError: 83040:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5741.850639] LustreError: lustre-MDT0000-mdc-ffff952b47105800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5741.868702] LustreError: 83083:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff952b47105800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff952b50821700) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5741.881160] LustreError: 83083:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5742.254601] LustreError: 83096:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5745.149120] LustreError: lustre-MDT0000-mdc-ffff952b47105800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5745.247295] LustreError: 83115:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5745.263626] LustreError: 83115:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5746.256479] LustreError: 83096:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5746.263860] LustreError: 83096:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff952b47105800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5746.361433] LustreError: 83121:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5746.381747] LustreError: 83121:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5748.719072] LustreError: 83144:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5748.733066] LustreError: 83144:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5752.370706] LustreError: 83170:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5755.375292] LustreError: lustre-MDT0000-mdc-ffff952b47105800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5755.399854] LustreError: 83184:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff952b47105800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff952b44868b00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5756.439567] LustreError: 83170:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5763.510096] LustreError: lustre-MDT0000-mdc-ffff952b47105800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5763.992483] LustreError: 83226:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5766.828926] LustreError: lustre-MDT0000-mdc-ffff952b47105800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5766.855422] LustreError: 83240:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff952b47105800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff952b60486c00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5766.891467] LustreError: 83240:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 2 previous similar messages
[ 5768.065238] LustreError: 83226:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5771.176096] LustreError: lustre-MDT0000-mdc-ffff952b47105800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5771.272246] LustreError: 83273:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5771.283642] LustreError: 83273:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5772.240505] LustreError: 83253:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff952b47105800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5780.270464] LustreError: 83324:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5780.278392] LustreError: 83324:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5784.000812] LustreError: lustre-MDT0000-mdc-ffff952b47105800: operation ldlm_enqueue to node 192.168.204.147@tcp failed: rc = -107
[ 5784.020904] LustreError: Skipped 8 previous similar messages
[ 5784.040777] LustreError: lustre-MDT0000-mdc-ffff952b47105800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5784.100461] LustreError: 83346:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5784.106436] LustreError: 83346:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5784.351191] LustreError: 83324:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5784.361239] LustreError: 83324:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5784.376573] LustreError: 83324:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff952b47105800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5795.927473] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 15:35:36 (1787859336)
[ 5815.742411] LustreError: 70780:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5815.753663] LustreError: 70780:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5819.823170] LustreError: 70780:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5819.839593] LustreError: 70780:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5827.512011] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 15:36:08 (1787859368)
[ 5877.623203] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 15:36:58 (1787859418)
[ 5944.095126] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 15:38:04 (1787859484)
[ 5979.103541] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 15:38:40 (1787859520)
[ 5979.643632] Lustre: Mounted lustre-client - version 2.17.57_83_g3de4944
[ 6103.265316] Lustre: Unmounted lustre-client
[ 6109.876785] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 15:40:51 (1787859651)
[ 6110.482529] LustreError: 87650:0:(osc_request.c:3177:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 6113.332437] Lustre: lustre-OST0000-osc-ffff952b47105800: Connection to lustre-OST0000 (at 192.168.204.147@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6113.346817] Lustre: Skipped 17 previous similar messages
[ 6113.377154] LustreError: lustre-OST0000-osc-ffff952b47105800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6113.387954] LustreError: 87737:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff952b47105800: namespace resource [0x240000400:0x3d8d:0x0].0x0 (ffff952b44868800) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6113.444957] LustreError: 87737:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 6113.655600] LustreError: 87650:0:(osc_request.c:3177:osc_build_rpc()) cfs_fail_timeout interrupted
[ 6113.663600] Lustre: 2348:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.147@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 6113.679828] Lustre: lustre-OST0000-osc-ffff952b47105800: Connection restored to 192.168.204.147@tcp (at 192.168.204.147@tcp)
[ 6113.703872] Lustre: Skipped 18 previous similar messages
[ 6120.582646] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 15:41:01 (1787859661)
[ 6178.783190] Lustre: 88317:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787859666/real 1787859666]  req@ffff952b52073480 x1874700397068416/t0(0) o101->lustre-MDT0000-mdc-ffff952b47105800@192.168.204.147@tcp:12/10 lens 328/344 e 0 to 1 dl 1787859721 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 6178.822321] Lustre: 88317:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 19 previous similar messages
[ 6185.883018] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 15:42:07 (1787859727)
[ 6187.529095] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 6189.256295] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 15:42:10 (1787859730)
[ 6256.658471] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6258.682444] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 15:43:19 (1787859799)
[ 6318.043757] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 15:44:19 (1787859859)
[ 6321.283677] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6323.789554] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 15:44:24 (1787859864)
[ 6325.777761] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6327.817480] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 15:44:29 (1787859869)
[ 6368.996687] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 15:45:10 (1787859910)
[ 6380.865629] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6408.159248] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 192.168.204.147@tcp) was lost; in progress operations using this service will fail
[ 6408.191358] LustreError: Skipped 2 previous similar messages
[ 6418.467582] Lustre: Evicted from MGS (at 192.168.204.147@tcp) after server handle changed from 0xce1e7d5b56b2f28a to 0xce1e7d5b56b30d3a
[ 6418.479137] Lustre: Skipped 2 previous similar messages
[ 6422.785247] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6424.118195] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6434.925773] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 15:46:16 (1787859976)
[ 6437.911772] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6440.128016] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 15:46:21 (1787859981)
[ 6459.375779] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 192.168.204.147@tcp) was lost; in progress operations using this service will fail
[ 6459.405166] Lustre: Evicted from MGS (at 192.168.204.147@tcp) after server handle changed from 0xce1e7d5b56b30d3a to 0xce1e7d5b56b30f47
[ 6473.521906] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 15:46:54 (1787860014)
[ 6521.771100] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 15:47:43 (1787860063)
[ 6528.300887] LustreError: lustre-OST0000-osc-ffff952b47105800: operation ost_setattr to node 192.168.204.147@tcp failed: rc = -19
[ 6528.324838] LustreError: Skipped 1 previous similar message
[ 6580.882093] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6584.153904] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6669.282718] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 192.168.204.147@tcp) was lost; in progress operations using this service will fail
[ 6669.300635] LustreError: Skipped 1 previous similar message
[ 6679.593679] Lustre: Evicted from MGS (at 192.168.204.147@tcp) after server handle changed from 0xce1e7d5b56b312ce to 0xce1e7d5b56b32367
[ 6679.611036] Lustre: Skipped 1 previous similar message
[ 6687.817977] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6689.462225] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6715.370178] Lustre: lustre-MDT0000-mdc-ffff952b47105800: Connection to lustre-MDT0000 (at 192.168.204.147@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6715.391720] Lustre: Skipped 8 previous similar messages
[ 6715.489385] Lustre: lustre-MDT0000-mdc-ffff952b47105800: Connection restored to 192.168.204.147@tcp (at 192.168.204.147@tcp)
[ 6715.502336] Lustre: Skipped 11 previous similar messages
[ 6724.908579] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6726.465923] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6863.372550] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 15:53:25 (1787860405)
[ 6930.771319] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6934.614213] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 7192.713517] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 15:58:53 (1787860733)
[ 7376.876295] Lustre: lustre-MDT0000-mdc-ffff952b47105800: Connection to lustre-MDT0000 (at 192.168.204.147@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7376.893532] Lustre: Skipped 1 previous similar message
[ 7376.898566] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 192.168.204.147@tcp) was lost; in progress operations using this service will fail
[ 7376.912467] LustreError: Skipped 1 previous similar message
[ 7376.925351] Lustre: Evicted from MGS (at 192.168.204.147@tcp) after server handle changed from 0xce1e7d5b56b3270a to 0xce1e7d5b56c29071
[ 7376.935840] Lustre: Skipped 1 previous similar message
[ 7376.940920] Lustre: MGC192.168.204.147@tcp: Connection restored to 192.168.204.147@tcp (at 192.168.204.147@tcp)
[ 7376.950378] Lustre: Skipped 1 previous similar message
[ 7404.512815] Lustre: 2347:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787860909/real 1787860909]  req@ffff952b71153480 x1874700414511232/t0(0) o400->lustre-MDT0000-mdc-ffff952b47105800@192.168.204.147@tcp:12/10 lens 224/224 e 0 to 1 dl 1787860947 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7404.561833] Lustre: 2347:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 25 previous similar messages
[ 7411.921806] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 16:02:33 (1787860953)
[ 7413.654779] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7415.463493] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 16:02:36 (1787860956)
[ 7417.746865] LustreError: lustre-MDT0000-mdc-ffff952b47105800: operation ldlm_enqueue to node 192.168.204.147@tcp failed: rc = -107
[ 7417.760941] LustreError: Skipped 177 previous similar messages
[ 7417.774687] LustreError: lustre-MDT0000-mdc-ffff952b47105800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7425.239550] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 16:02:46 (1787860966)
[ 7592.695213] LustreError: lustre-OST0000-osc-ffff952b47105800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7594.540657] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 16:05:36 (1787861136)
[ 7619.552922] Lustre: 2347:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787861142/real 1787861142]  req@ffff952b46c1cf00 x1874700414531840/t0(0) o4->lustre-OST0000-osc-ffff952b47105800@192.168.204.147@tcp:6/4 lens 4584/448 e 0 to 1 dl 1787861162 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7619.589201] Lustre: 2347:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 7635.719241] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 16:06:17 (1787861177)
[ 7637.020056] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7639.444914] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 16:06:20 (1787861180)
[ 7641.334269] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7643.143593] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 16:06:24 (1787861184)
[ 7677.452322] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 16:06:59 (1787861219)
[ 7715.823487] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 192.168.204.147@tcp) was lost; in progress operations using this service will fail
[ 7715.866854] Lustre: Evicted from MGS (at 192.168.204.147@tcp) after server handle changed from 0xce1e7d5b56c29071 to 0xce1e7d5b56c2baf6
[ 7715.879111] Lustre: Skipped 1 previous similar message
[ 7733.951512] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 16:07:55 (1787861275)
[ 7735.511559] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7737.416728] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 16:07:58 (1787861278)
[ 7739.118604] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7740.944418] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 16:08:02 (1787861282)
[ 7751.380772] Lustre: Unmounted lustre-client
[ 7753.527471] Lustre: Mounted lustre-client - version 2.17.57_83_g3de4944
[ 7756.559948] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7774.175362] Lustre: 2349:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787861301/real 1787861301]  req@ffff952b69323840 x1874700414718464/t0(0) o400->MGC192.168.204.147@tcp@192.168.204.147@tcp:26/25 lens 224/224 e 0 to 1 dl 1787861317 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7774.209907] Lustre: 2349:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 7785.519756] Lustre: 106785:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.147@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 7798.708394] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2