[ 2024.875853] LustreError: lustre-OST0000-osc-ffff936dc8d58800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2024.893840] Lustre: 2354:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.117@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 2032.503578] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 15:00:02 (1789585202)
[ 2034.877889] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 2037.126876] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 15:00:06 (1789585206)
[ 2039.122205] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 2041.178747] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 15:00:10 (1789585210)
[ 2063.327463] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 192.168.204.117@tcp) was lost; in progress operations using this service will fail
[ 2073.530872] Lustre: Evicted from MGS (at 192.168.204.117@tcp) after server handle changed from 0xb5f0d9f133dbe92d to 0xb5f0d9f133dc04d2
[ 2170.146259] LustreError: lustre-MDT0000-mdc-ffff936dc8d58800: operation ldlm_enqueue to node 192.168.204.117@tcp failed: rc = -19
[ 2170.168031] LustreError: Skipped 2 previous similar messages
[ 2191.328679] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 192.168.204.117@tcp) was lost; in progress operations using this service will fail
[ 2191.370228] Lustre: Evicted from MGS (at 192.168.204.117@tcp) after server handle changed from 0xb5f0d9f133dc04d2 to 0xb5f0d9f133ddd36c
[ 2193.002324] Lustre: 15951:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.117@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2219.247356] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 15:03:08 (1789585388)
[ 2219.826337] Lustre: *** cfs_fail_loc=305, val=0***
[ 2219.828250] Lustre: Skipped 2 previous similar messages
[ 2239.301826] LustreError: lustre-OST0000-osc-ffff936dc8d58800: operation ost_connect to node 192.168.204.117@tcp failed: rc = -75
[ 2261.983914] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 192.168.204.117@tcp) was lost; in progress operations using this service will fail
[ 2266.101145] Lustre: lustre-MDT0000-mdc-ffff936dc8d58800: Connection to lustre-MDT0000 (at 192.168.204.117@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2266.123144] Lustre: Skipped 11 previous similar messages
[ 2266.234266] Lustre: lustre-MDT0000-mdc-ffff936dc8d58800: Connection restored to 192.168.204.117@tcp (at 192.168.204.117@tcp)
[ 2266.243472] Lustre: Skipped 13 previous similar messages
[ 2272.950231] Lustre: Evicted from MGS (at 192.168.204.117@tcp) after server handle changed from 0xb5f0d9f133ddd36c to 0xb5f0d9f133dddc25
[ 2277.605560] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2279.660274] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2287.987387] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 15:04:17 (1789585457)
[ 2303.979685] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 192.168.204.117@tcp) was lost; in progress operations using this service will fail
[ 2304.003103] Lustre: Evicted from MGS (at 192.168.204.117@tcp) after server handle changed from 0xb5f0d9f133dddc25 to 0xb5f0d9f133dddec5
[ 2304.047646] LustreError: lustre-MDT0000-mdc-ffff936dc8d58800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2325.772873] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 15:04:55 (1789585495)
[ 2361.765028] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 15:05:31 (1789585531)
[ 2375.642575] LustreError: lustre-MDT0000-mdc-ffff936dc8d58800: operation mds_close to node 192.168.204.117@tcp failed: rc = -19
[ 2395.624807] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 192.168.204.117@tcp) was lost; in progress operations using this service will fail
[ 2395.656088] Lustre: Evicted from MGS (at 192.168.204.117@tcp) after server handle changed from 0xb5f0d9f133dddec5 to 0xb5f0d9f133de2279
[ 2397.228649] Lustre: 15951:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.117@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2422.307971] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2424.865560] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2512.847282] Lustre: 2356:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789585668/real 1789585668]  req@ffff936dc32d9c00 x1876514045326848/t0(0) o400->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 224/224 e 0 to 1 dl 1789585684 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2512.890794] Lustre: 2356:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 12 previous similar messages
[ 2512.911143] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 192.168.204.117@tcp) was lost; in progress operations using this service will fail
[ 2512.942850] Lustre: Evicted from MGS (at 192.168.204.117@tcp) after server handle changed from 0xb5f0d9f133de2279 to 0xb5f0d9f133dfb061
[ 2541.685295] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2544.067662] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2629.614205] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 192.168.204.117@tcp) was lost; in progress operations using this service will fail
[ 2629.642067] Lustre: Evicted from MGS (at 192.168.204.117@tcp) after server handle changed from 0xb5f0d9f133dfb061 to 0xb5f0d9f133e136af
[ 2646.056336] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2648.093930] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2676.394876] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 15:10:46 (1789585846)
[ 2680.105040] LustreError: lustre-MDT0000-mdc-ffff936dc8d58800: operation mds_close to node 192.168.204.117@tcp failed: rc = -19
[ 2680.110026] LustreError: Skipped 3 previous similar messages
[ 2696.223279] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 192.168.204.117@tcp) was lost; in progress operations using this service will fail
[ 2706.482864] Lustre: Evicted from MGS (at 192.168.204.117@tcp) after server handle changed from 0xb5f0d9f133e136af to 0xb5f0d9f133e23a7a
[ 2716.288099] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2745.500782] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2782.597343] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2825.073786] Lustre: 15951:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.117@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2835.709503] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2876.908081] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 192.168.204.117@tcp) was lost; in progress operations using this service will fail
[ 2876.923709] LustreError: Skipped 3 previous similar messages
[ 2876.954196] Lustre: Evicted from MGS (at 192.168.204.117@tcp) after server handle changed from 0xb5f0d9f133e26f94 to 0xb5f0d9f133e318b3
[ 2876.967816] Lustre: Skipped 3 previous similar messages
[ 2876.981364] Lustre: MGC192.168.204.117@tcp: Connection restored to 192.168.204.117@tcp (at 192.168.204.117@tcp)
[ 2876.989753] Lustre: Skipped 16 previous similar messages
[ 2879.031945] Lustre: 15951:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.117@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2897.504502] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2925.581254] Lustre: lustre-MDT0000-mdc-ffff936dc8d58800: Connection to lustre-MDT0000 (at 192.168.204.117@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2925.595947] Lustre: Skipped 9 previous similar messages
[ 2945.148198] Lustre: 15951:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.117@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2961.835732] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 3047.595920] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 15:16:57 (1789586217)
[ 3096.224211] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3098.125243] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3391.985553] LustreError: lustre-OST0000-osc-ffff936dc8d58800: operation ldlm_enqueue to node 192.168.204.117@tcp failed: rc = -19
[ 3391.992666] LustreError: Skipped 11 previous similar messages
[ 3429.433663] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3431.526542] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3724.730339] Lustre: lustre-OST0000-osc-ffff936dc8d58800: Connection to lustre-OST0000 (at 192.168.204.117@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3724.744967] Lustre: Skipped 3 previous similar messages
[ 3746.029237] Lustre: lustre-OST0000-osc-ffff936dc8d58800: Connection restored to 192.168.204.117@tcp (at 192.168.204.117@tcp)
[ 3746.045090] Lustre: Skipped 6 previous similar messages
[ 3764.436982] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3767.047854] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4020.407756] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 15:33:10 (1789587190)
[ 4038.111384] Lustre: 50617:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789587193/real 1789587193]  req@ffff936dc34cdf80 x1876514067360384/t0(0) o101->lustre-MDT0000-mdc-ffff936dc8d58800@192.168.204.117@tcp:12/10 lens 576/1152 e 0 to 1 dl 1789587209 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 4038.153473] Lustre: 50617:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
[ 4048.714553] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 15:33:38 (1789587218)
[ 4075.495153] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 15:34:05 (1789587245)
[ 4102.474255] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 15:34:32 (1789587272)
[ 4102.885828] Lustre: Mounted lustre-client - version 2.17.58_40_g81e4dae
[ 4133.855221] Lustre: 2354:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789587289/real 1789587289]  req@ffff936dc355d500 x1876514067380480/t0(0) o400->lustre-MDT0000-mdc-ffff936dcc86a000@192.168.204.117@tcp:12/10 lens 224/224 e 0 to 1 dl 1789587305 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4133.907453] Lustre: 2354:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 4133.913842] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 192.168.204.117@tcp) was lost; in progress operations using this service will fail
[ 4133.920770] LustreError: Skipped 2 previous similar messages
[ 4144.140428] Lustre: Evicted from MGS (at 192.168.204.117@tcp) after server handle changed from 0xb5f0d9f133e45ffa to 0xb5f0d9f133f98952
[ 4144.152255] Lustre: Skipped 2 previous similar messages
[ 4151.310798] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4152.993295] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4155.021516] Lustre: Unmounted lustre-client
[ 4162.507339] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 15:35:31 (1789587331)
[ 4285.407183] Lustre: 2355:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789587440/real 1789587440]  req@ffff936dc32db100 x1876514067413376/t0(0) o4->lustre-OST0000-osc-ffff936dc8d58800@192.168.204.117@tcp:6/4 lens 488/448 e 0 to 1 dl 1789587456 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4285.407183] Lustre: 2354:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789587440/real 1789587440]  req@ffff936dc4813b80 x1876514067413888/t0(0) o4->lustre-OST0000-osc-ffff936dc8d58800@192.168.204.117@tcp:6/4 lens 488/448 e 0 to 1 dl 1789587456 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4285.407202] Lustre: 2355:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 54 previous similar messages
[ 4285.466814] Lustre: 2354:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 60 previous similar messages
[ 4334.563490] Lustre: lustre-OST0000-osc-ffff936dc8d58800: Connection to lustre-OST0000 (at 192.168.204.117@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4334.585167] Lustre: Skipped 14 previous similar messages
[ 4349.983890] Lustre: lustre-OST0000-osc-ffff936dc8d58800: Connection restored to 192.168.204.117@tcp (at 192.168.204.117@tcp)
[ 4349.998063] Lustre: Skipped 16 previous similar messages
[ 4462.561700] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 15:40:32 (1789587632)
[ 4513.205549] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 15:41:22 (1789587682)
[ 4516.996923] Lustre: Unmounted lustre-client
[ 4547.489725] Lustre: Mounted lustre-client - version 2.17.58_40_g81e4dae
[ 4555.624814] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 15:42:05 (1789587725)
[ 4556.400100] LustreError: 56544:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4557.464881] LustreError: 56544:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4557.633719] Lustre: *** cfs_fail_loc=305, val=0***
[ 4582.023152] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 15:42:31 (1789587751)
[ 4582.688081] Lustre: Mounted lustre-client - version 2.17.58_40_g81e4dae
[ 4586.107667] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4596.464150] Lustre: Unmounted lustre-client
[ 4603.597620] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 15:42:53 (1789587773)
[ 4695.286054] LustreError: lustre-MDT0000-mdc-ffff936dc9d30800: operation mds_reint to node 192.168.204.117@tcp failed: rc = -19
[ 4695.305076] LustreError: Skipped 1 previous similar message
[ 4716.836724] Lustre: 2355:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789587872/real 1789587872]  req@ffff936dc6f80380 x1876514070100864/t0(0) o400->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 224/224 e 0 to 1 dl 1789587888 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4716.871626] Lustre: 2355:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 99 previous similar messages
[ 4716.878399] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 192.168.204.117@tcp) was lost; in progress operations using this service will fail
[ 4727.141649] Lustre: Evicted from MGS (at 192.168.204.117@tcp) after server handle changed from 0xb5f0d9f133f9a15b to 0xb5f0d9f133fc5b4e
[ 4918.344507] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 15:48:07 (1789588087)
[ 4924.195274] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4941.294798] Lustre: lustre-MDT0000-mdc-ffff936dc9d30800: Connection to lustre-MDT0000 (at 192.168.204.117@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4941.296645] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 192.168.204.117@tcp) was lost; in progress operations using this service will fail
[ 4941.309049] Lustre: Skipped 10 previous similar messages
[ 4941.334285] Lustre: Evicted from MGS (at 192.168.204.117@tcp) after server handle changed from 0xb5f0d9f133fc5b4e to 0xb5f0d9f134011d2b
[ 4941.418057] LustreError: lustre-MDT0000-mdc-ffff936dc9d30800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4959.032290] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 15:48:48 (1789588128)
[ 4959.626306] Lustre: Mounted lustre-client - version 2.17.58_40_g81e4dae
[ 4967.071144] LustreError: lustre-OST0000-osc-ffff936dc9d30800: operation ldlm_enqueue to node 192.168.204.117@tcp failed: rc = -107
[ 4967.091772] LustreError: lustre-OST0000-osc-ffff936dc9d30800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4977.175977] Lustre: lustre-OST0000-osc-ffff936dca4c9000: Connection restored to 192.168.204.117@tcp (at 192.168.204.117@tcp)
[ 4977.211203] Lustre: Skipped 12 previous similar messages
[ 4978.672784] Lustre: Unmounted lustre-client
[ 4987.767550] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 15:49:16 (1789588156)
[ 4995.513480] LustreError: lustre-MDT0000-mdc-ffff936dc9d30800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4995.532649] LustreError: 60957:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 5007.389256] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 15:49:36 (1789588176)
[ 5007.630413] LustreError: 61641:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 5012.960584] LustreError: 61641:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=0
[ 5012.974743] LustreError: 61641:0:(import.c:756:ptlrpc_connect_import_locked()) already connected
[ 5013.554787] LustreError: lustre-MDT0000-mdc-ffff936dc9d30800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5013.588472] LustreError: 61664:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 5013.735472] LustreError: 61669:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5015.006865] LustreError: 61675:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5015.031357] LustreError: 61675:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5017.613364] LustreError: 61697:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5017.635582] LustreError: 61697:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5022.593993] LustreError: 61742:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5022.616171] LustreError: 61742:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 5032.730110] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 15:50:02 (1789588202)
[ 5075.392936] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5077.213250] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5088.856752] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 15:50:58 (1789588258)
[ 5132.264238] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5134.441353] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5146.024743] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 15:51:55 (1789588315)
[ 5214.300877] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5216.201965] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5225.814569] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 15:53:15 (1789588395)
[ 5267.552564] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5269.981192] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5276.606948] Lustre: Unmounted lustre-client
[ 5331.771225] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5334.158661] Lustre: Mounted lustre-client - version 2.17.58_40_g81e4dae
[ 5341.920533] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 15:55:11 (1789588511)
[ 5344.806520] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5347.106803] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 15:55:16 (1789588516)
[ 5389.066632] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 15:55:58 (1789588558)
[ 5390.852510] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5393.635510] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 15:56:02 (1789588562)
[ 5395.607598] Lustre: *** cfs_fail_loc=805, val=0***
[ 5395.664642] Lustre: Mounted lustre-client - version 2.17.58_40_g81e4dae
[ 5403.753845] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5422.559628] Lustre: 2354:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789588577/real 1789588577]  req@ffff936dc5769f80 x1876514072144256/t0(0) o400->lustre-MDT0000-mdc-ffff936dc9683800@192.168.204.117@tcp:12/10 lens 224/224 e 0 to 1 dl 1789588593 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5422.614526] Lustre: 2354:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5422.627659] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 192.168.204.117@tcp) was lost; in progress operations using this service will fail
[ 5432.812922] Lustre: Evicted from MGS (at 192.168.204.117@tcp) after server handle changed from 0xb5f0d9f1340127c7 to 0xb5f0d9f1340129e2
[ 5447.145672] LustreError: 2352:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff936dc8eb5000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff936dc726da00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5449.513402] Lustre: Unmounted lustre-client
[ 5457.785458] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 15:57:07 (1789588627)
[ 5502.254757] LustreError: 2352:0:(client.c:3449:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff936dc6a1f800 x1876514072156672/t90194313221(90194313221) o101->lustre-MDT0000-mdc-ffff936dc9683800@192.168.204.117@tcp:12/10 lens 648/608 e 0 to 0 dl 1789588689 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5508.781596] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5510.403324] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5520.393820] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 15:58:09 (1789588689)
[ 5522.133067] LustreError: lustre-OST0000-osc-ffff936dc9683800: operation ost_write to node 192.168.204.117@tcp failed: rc = -107
[ 5522.141612] LustreError: Skipped 2 previous similar messages
[ 5522.165287] LustreError: lustre-OST0000-osc-ffff936dc9683800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5522.190061] Lustre: 2356:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.117@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5522.242159] LustreError: 72841:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff936dc9683800: namespace resource [0x240000400:0x3e42:0x0].0x0 (ffff936dc726d100) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5522.273907] LustreError: 72841:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5531.127953] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 15:58:20 (1789588700)
[ 5533.019714] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5535.058497] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 15:58:24 (1789588704)
[ 5536.681596] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5538.496608] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 15:58:28 (1789588708)
[ 5540.056462] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5541.680637] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 15:58:31 (1789588711)
[ 5543.387200] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5545.095785] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 15:58:34 (1789588714)
[ 5546.619806] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5548.406055] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 15:58:38 (1789588718)
[ 5550.044463] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5551.958679] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 15:58:41 (1789588721)
[ 5553.987884] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5556.211869] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 15:58:45 (1789588725)
[ 5558.397460] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5560.437744] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 15:58:50 (1789588730)
[ 5561.862065] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5563.998799] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 15:58:53 (1789588733)
[ 5565.715661] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5567.649293] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 15:58:57 (1789588737)
[ 5569.085490] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5570.963638] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 15:59:00 (1789588740)
[ 5573.543619] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5575.537833] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 15:59:05 (1789588745)
[ 5595.615356] Lustre: lustre-MDT0000-mdc-ffff936dc9683800: Connection to lustre-MDT0000 (at 192.168.204.117@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5595.636988] Lustre: Skipped 14 previous similar messages
[ 5604.856887] Lustre: MGC192.168.204.117@tcp: Connection restored to 192.168.204.117@tcp (at 192.168.204.117@tcp)
[ 5604.878024] Lustre: Skipped 12 previous similar messages
[ 5611.145181] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 15:59:40 (1789588780)
[ 5642.613256] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 16:00:12 (1789588812)
[ 5643.424885] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5656.851665] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 16:00:25 (1789588825)
[ 5657.495369] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5670.157213] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 16:00:39 (1789588839)
[ 5670.608676] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5681.996898] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 16:00:51 (1789588851)
[ 5682.426131] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5692.907178] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 16:01:02 (1789588862)
[ 5693.478366] Lustre: *** cfs_fail_loc=510, val=3***
[ 5703.979977] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 16:01:13 (1789588873)
[ 5704.600710] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5715.355594] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 16:01:25 (1789588885)
[ 5715.830726] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5783.481488] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 16:02:32 (1789588952)
[ 5783.754172] LustreError: 83064:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5787.139768] LustreError: lustre-MDT0000-mdc-ffff936dc9683800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5787.160327] LustreError: 83078:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff936dc9683800: inode [0x20000a042:0x10:0x0] mdc close failed: rc = -108
[ 5787.199406] LustreError: 83078:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff936dc9683800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff936dc6f9c000) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5787.210057] LustreError: 83083:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5787.219484] LustreError: 83083:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5787.759168] LustreError: 83064:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5789.957872] LustreError: 83094:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5793.147657] LustreError: lustre-MDT0000-mdc-ffff936dc9683800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5793.192230] LustreError: 83108:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff936dc9683800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff936dc726d300) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5793.248777] LustreError: 83108:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5793.984126] LustreError: 83094:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5801.548604] LustreError: lustre-MDT0000-mdc-ffff936dc9683800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5802.126978] LustreError: 83149:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5805.066514] LustreError: lustre-MDT0000-mdc-ffff936dc9683800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5805.192466] LustreError: 83169:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5806.199248] LustreError: 83149:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5806.208704] LustreError: 83149:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff936dc9683800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5807.475636] LustreError: 83186:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5807.494733] LustreError: 83186:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 12 previous similar messages
[ 5812.474827] LustreError: 83224:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5815.487897] LustreError: lustre-MDT0000-mdc-ffff936dc9683800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5815.520191] LustreError: 83238:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff936dc9683800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff936dc6f4b300) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5815.543271] LustreError: 83238:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5816.479143] LustreError: 83224:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5823.669563] LustreError: lustre-MDT0000-mdc-ffff936dc9683800: operation ldlm_enqueue to node 192.168.204.117@tcp failed: rc = -107
[ 5823.680284] LustreError: Skipped 5 previous similar messages
[ 5823.700158] LustreError: lustre-MDT0000-mdc-ffff936dc9683800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5824.157597] LustreError: 83280:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5827.222117] LustreError: lustre-MDT0000-mdc-ffff936dc9683800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5827.236635] LustreError: 83294:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff936dc9683800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff936dc6f9c600) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5827.248258] LustreError: 83294:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5828.247669] LustreError: 83280:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5831.322938] LustreError: lustre-MDT0000-mdc-ffff936dc9683800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5831.370936] LustreError: 83327:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5831.379349] LustreError: 83327:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5832.472689] LustreError: 83307:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff936dc9683800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5840.405756] LustreError: 83379:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5840.416945] LustreError: 83379:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5844.496083] LustreError: 83379:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5844.517205] LustreError: 83379:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5844.579267] LustreError: lustre-MDT0000-mdc-ffff936dc9683800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5844.608023] LustreError: 83379:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff936dc9683800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -5
[ 5853.643975] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 16:03:42 (1789589022)
[ 5873.266413] LustreError: 68297:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5877.327757] LustreError: 68297:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5885.339948] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 16:04:15 (1789589055)
[ 5936.464740] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 16:05:06 (1789589106)
[ 6003.250680] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 16:06:12 (1789589172)
[ 6039.915517] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 16:06:49 (1789589209)
[ 6041.195378] Lustre: Mounted lustre-client - version 2.17.58_40_g81e4dae
[ 6167.355947] Lustre: Unmounted lustre-client
[ 6177.627207] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 16:09:06 (1789589346)
[ 6179.136356] LustreError: 87662:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 6181.813623] LustreError: lustre-OST0000-osc-ffff936dc9683800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6181.845378] LustreError: 87749:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff936dc9683800: namespace resource [0x240000400:0x3e6d:0x0].0x0 (ffff936dc8ee7e00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6181.898344] LustreError: 87749:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 2 previous similar messages
[ 6182.175136] LustreError: 87662:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout interrupted
[ 6182.190880] Lustre: 2353:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.117@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 6190.522944] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 16:09:19 (1789589359)
[ 6248.415286] Lustre: 88330:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789589364/real 1789589364]  req@ffff936dc576ad80 x1876514072323712/t0(0) o101->lustre-MDT0000-mdc-ffff936dc9683800@192.168.204.117@tcp:12/10 lens 328/344 e 0 to 1 dl 1789589419 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 6248.451764] Lustre: 88330:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 21 previous similar messages
[ 6248.463836] Lustre: lustre-MDT0000-mdc-ffff936dc9683800: Connection to lustre-MDT0000 (at 192.168.204.117@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6248.493281] Lustre: Skipped 14 previous similar messages
[ 6248.526871] Lustre: lustre-MDT0000-mdc-ffff936dc9683800: Connection restored to 192.168.204.117@tcp (at 192.168.204.117@tcp)
[ 6248.537684] Lustre: Skipped 14 previous similar messages
[ 6256.638872] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 16:10:26 (1789589426)
[ 6258.371690] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 6260.309148] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 16:10:29 (1789589429)
[ 6324.402445] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6326.351515] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 16:11:36 (1789589496)
[ 6385.708223] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 16:12:35 (1789589555)
[ 6388.721821] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6391.028712] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 16:12:40 (1789589560)
[ 6393.503830] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6395.288898] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 16:12:45 (1789589565)
[ 6436.268370] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 16:13:25 (1789589605)
[ 6448.896301] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6476.767279] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 192.168.204.117@tcp) was lost; in progress operations using this service will fail
[ 6476.773443] LustreError: Skipped 2 previous similar messages
[ 6486.009529] Lustre: Evicted from MGS (at 192.168.204.117@tcp) after server handle changed from 0xb5f0d9f134013629 to 0xb5f0d9f134014e24
[ 6486.033882] Lustre: Skipped 2 previous similar messages
[ 6504.267834] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6506.086888] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6517.107339] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 16:14:46 (1789589686)
[ 6519.742539] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6521.668394] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 16:14:51 (1789589691)
[ 6538.226088] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 192.168.204.117@tcp) was lost; in progress operations using this service will fail
[ 6538.266631] Lustre: Evicted from MGS (at 192.168.204.117@tcp) after server handle changed from 0xb5f0d9f134014e24 to 0xb5f0d9f1340152d8
[ 6549.342895] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 16:15:19 (1789589719)
[ 6601.277031] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 16:16:11 (1789589771)
[ 6611.053157] LustreError: lustre-OST0000-osc-ffff936dc9683800: operation ost_setattr to node 192.168.204.117@tcp failed: rc = -107
[ 6611.071388] LustreError: Skipped 4 previous similar messages
[ 6670.751854] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6675.802675] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6758.879328] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 192.168.204.117@tcp) was lost; in progress operations using this service will fail
[ 6758.895132] LustreError: Skipped 1 previous similar message
[ 6769.135597] Lustre: Evicted from MGS (at 192.168.204.117@tcp) after server handle changed from 0xb5f0d9f1340154e5 to 0xb5f0d9f1340163c5
[ 6769.150980] Lustre: Skipped 1 previous similar message
[ 6786.733471] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6789.258307] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6837.047524] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6839.030409] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6854.626549] Lustre: 2353:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789589965/real 1789589965]  req@ffff936de6442a00 x1876514078851072/t0(0) o400->lustre-MDT0000-mdc-ffff936dc9683800@192.168.204.117@tcp:12/10 lens 224/224 e 0 to 1 dl 1789590025 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6854.673869] Lustre: 2353:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 23 previous similar messages
[ 6958.869555] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 16:22:09 (1789590129)
[ 6969.396870] Lustre: lustre-OST0000-osc-ffff936dc9683800: Connection to lustre-OST0000 (at 192.168.204.117@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6969.419534] Lustre: Skipped 8 previous similar messages
[ 7008.839503] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 7011.098958] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 7244.764092] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 16:26:54 (1789590414)
[ 7398.896593] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 192.168.204.117@tcp) was lost; in progress operations using this service will fail
[ 7398.925487] LustreError: Skipped 1 previous similar message
[ 7398.944533] Lustre: Evicted from MGS (at 192.168.204.117@tcp) after server handle changed from 0xb5f0d9f13401673e to 0xb5f0d9f134110ac0
[ 7398.957585] Lustre: Skipped 1 previous similar message
[ 7398.964521] Lustre: MGC192.168.204.117@tcp: Connection restored to 192.168.204.117@tcp (at 192.168.204.117@tcp)
[ 7398.970436] Lustre: Skipped 12 previous similar messages
[ 7433.888281] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 16:30:03 (1789590603)
[ 7435.960947] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7437.328852] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 16:30:07 (1789590607)
[ 7439.448322] LustreError: lustre-MDT0000-mdc-ffff936dc9683800: operation ldlm_enqueue to node 192.168.204.117@tcp failed: rc = -107
[ 7439.464664] LustreError: Skipped 54 previous similar messages
[ 7439.482467] LustreError: lustre-MDT0000-mdc-ffff936dc9683800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7448.361622] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 16:30:18 (1789590618)
[ 7615.464276] LustreError: lustre-OST0000-osc-ffff936dc9683800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7618.011315] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 16:33:07 (1789590787)
[ 7643.871232] Lustre: 2354:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789590795/real 1789590795]  req@ffff936dd5e82a00 x1876514090029824/t0(0) o4->lustre-OST0000-osc-ffff936dc9683800@192.168.204.117@tcp:6/4 lens 4584/448 e 0 to 1 dl 1789590815 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7643.913686] Lustre: 2354:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
[ 7643.923810] Lustre: lustre-OST0000-osc-ffff936dc9683800: Connection to lustre-OST0000 (at 192.168.204.117@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7643.937358] Lustre: Skipped 3 previous similar messages
[ 7662.462573] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 16:33:52 (1789590832)
[ 7663.741708] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7665.934643] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 16:33:55 (1789590835)
[ 7667.826528] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7670.016257] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 16:33:59 (1789590839)
[ 7706.245703] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 16:34:35 (1789590875)
[ 7746.545044] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 192.168.204.117@tcp) was lost; in progress operations using this service will fail
[ 7746.571100] Lustre: Evicted from MGS (at 192.168.204.117@tcp) after server handle changed from 0xb5f0d9f134110ac0 to 0xb5f0d9f134113393
[ 7746.580284] Lustre: Skipped 1 previous similar message
[ 7766.101914] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 16:35:35 (1789590935)
[ 7768.117889] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7770.272172] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 16:35:39 (1789590939)
[ 7771.942197] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7774.163410] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 16:35:43 (1789590943)
[ 7787.327266] Lustre: Unmounted lustre-client
[ 7789.779474] Lustre: Mounted lustre-client - version 2.17.58_40_g81e4dae
[ 7793.654150] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7834.656487] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2