[ 1935.584126] LustreError: 34764:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 1939.449066] LustreError: lustre-OST0000-osc-ffff91b511253800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1945.602828] LustreError: 34764:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 awake
[ 1945.613084] Lustre: 2357:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.134@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1945.614122] LustreError: 34764:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 1955.672193] LustreError: 34764:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 awake
[ 1955.680532] Lustre: 2355:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.134@tcp:/lustre/fid: [0x200000404:0x36:0x0]// may get corrupted (rc -108)
[ 1963.590944] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 22:33:56 (1790735636)
[ 1965.972624] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1967.707827] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 22:34:00 (1790735640)
[ 1969.481745] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1971.125468] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 22:34:03 (1790735643)
[ 1974.558058] LustreError: lustre-MDT0000-mdc-ffff91b511253800: operation mds_close to node 192.168.204.134@tcp failed: rc = -19
[ 1991.648491] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 192.168.204.134@tcp) was lost; in progress operations using this service will fail
[ 2001.838569] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0xb43ed9d3a2bd9574 to 0xb43ed9d3a2bdae41
[ 2104.606545] LustreError: lustre-MDT0000-mdc-ffff91b511253800: operation mds_close to node 192.168.204.134@tcp failed: rc = -19
[ 2104.616826] LustreError: Skipped 3 previous similar messages
[ 2123.744298] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 192.168.204.134@tcp) was lost; in progress operations using this service will fail
[ 2133.943456] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0xb43ed9d3a2bdae41 to 0xb43ed9d3a2bf94d6
[ 2148.609822] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 22:37:01 (1790735821)
[ 2149.024585] Lustre: *** cfs_fail_loc=305, val=0***
[ 2149.026446] Lustre: Skipped 4 previous similar messages
[ 2188.257088] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 192.168.204.134@tcp) was lost; in progress operations using this service will fail
[ 2198.523826] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0xb43ed9d3a2bf94d6 to 0xb43ed9d3a2bf99c9
[ 2199.669902] Lustre: 15926:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.134@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2215.579072] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2217.522558] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2226.730445] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 22:38:19 (1790735899)
[ 2244.598424] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 192.168.204.134@tcp) was lost; in progress operations using this service will fail
[ 2244.633220] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0xb43ed9d3a2bf99c9 to 0xb43ed9d3a2bf9fd4
[ 2244.741435] LustreError: lustre-MDT0000-mdc-ffff91b511253800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2268.125350] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 22:39:00 (1790735940)
[ 2285.558680] LustreError: lustre-OST0000-osc-ffff91b511253800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2302.205956] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 22:39:34 (1790735974)
[ 2316.232786] LustreError: lustre-MDT0000-mdc-ffff91b511253800: operation mds_reint to node 192.168.204.134@tcp failed: rc = -19
[ 2336.747039] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 192.168.204.134@tcp) was lost; in progress operations using this service will fail
[ 2336.773335] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0xb43ed9d3a2bf9fd4 to 0xb43ed9d3a2bfe95b
[ 2355.142734] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2358.318736] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2444.960403] Lustre: 2355:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790736102/real 1790736102]  req@ffff91b504a79880 x1877720419673728/t0(0) o400->MGC192.168.204.134@tcp@192.168.204.134@tcp:26/25 lens 224/224 e 0 to 1 dl 1790736118 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2445.013098] Lustre: 2355:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 15 previous similar messages
[ 2445.032335] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 192.168.204.134@tcp) was lost; in progress operations using this service will fail
[ 2445.103931] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0xb43ed9d3a2bfe95b to 0xb43ed9d3a2c1a373
[ 2471.452436] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2473.842131] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2539.057892] Lustre: lustre-MDT0000-mdc-ffff91b511253800: Connection to lustre-MDT0000 (at 192.168.204.134@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2539.074338] Lustre: Skipped 9 previous similar messages
[ 2556.896869] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 192.168.204.134@tcp) was lost; in progress operations using this service will fail
[ 2567.082907] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0xb43ed9d3a2c1a373 to 0xb43ed9d3a2c35630
[ 2567.098862] Lustre: MGC192.168.204.134@tcp: Connection restored to 192.168.204.134@tcp (at 192.168.204.134@tcp)
[ 2567.110768] Lustre: Skipped 15 previous similar messages
[ 2585.565888] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2587.976994] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2617.726668] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 22:44:50 (1790736290)
[ 2622.032267] LustreError: lustre-MDT0000-mdc-ffff91b511253800: operation ldlm_enqueue to node 192.168.204.134@tcp failed: rc = -19
[ 2622.044494] LustreError: Skipped 2 previous similar messages
[ 2643.040223] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 192.168.204.134@tcp) was lost; in progress operations using this service will fail
[ 2643.080676] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0xb43ed9d3a2c35630 to 0xb43ed9d3a2c427d5
[ 2644.721806] Lustre: 15926:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.134@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2658.961628] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2699.872989] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2746.436431] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2778.592429] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 192.168.204.134@tcp) was lost; in progress operations using this service will fail
[ 2778.607452] LustreError: Skipped 2 previous similar messages
[ 2778.624093] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0xb43ed9d3a2c45754 to 0xb43ed9d3a2c4a65a
[ 2778.642040] Lustre: Skipped 2 previous similar messages
[ 2780.662354] Lustre: 15926:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.134@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2795.703051] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2841.534978] Lustre: 15926:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.134@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2854.993786] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2919.758740] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 3005.995748] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 22:51:18 (1790736678)
[ 3060.463793] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3063.577591] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3351.060650] LustreError: lustre-OST0000-osc-ffff91b511253800: operation ldlm_enqueue to node 192.168.204.134@tcp failed: rc = -19
[ 3351.069166] LustreError: Skipped 10 previous similar messages
[ 3351.073830] Lustre: lustre-OST0000-osc-ffff91b511253800: Connection to lustre-OST0000 (at 192.168.204.134@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3351.084798] Lustre: Skipped 8 previous similar messages
[ 3374.867824] Lustre: lustre-OST0000-osc-ffff91b511253800: Connection restored to 192.168.204.134@tcp (at 192.168.204.134@tcp)
[ 3374.878360] Lustre: Skipped 16 previous similar messages
[ 3395.535967] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3398.088843] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3724.100619] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3726.049363] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3976.203986] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 23:07:29 (1790737649)
[ 3994.081745] Lustre: 50584:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790737651/real 1790737651]  req@ffff91b5059c9500 x1877720440905216/t0(0) o101->lustre-MDT0000-mdc-ffff91b511253800@192.168.204.134@tcp:12/10 lens 576/1152 e 0 to 1 dl 1790737667 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3994.121859] Lustre: 50584:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
[ 3994.137119] Lustre: lustre-MDT0000-mdc-ffff91b511253800: Connection to lustre-MDT0000 (at 192.168.204.134@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3994.156289] Lustre: Skipped 1 previous similar message
[ 3994.176861] Lustre: lustre-MDT0000-mdc-ffff91b511253800: Connection restored to 192.168.204.134@tcp (at 192.168.204.134@tcp)
[ 3994.183740] Lustre: Skipped 1 previous similar message
[ 4002.047511] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 23:07:54 (1790737674)
[ 4026.862357] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 23:08:19 (1790737699)
[ 4052.822928] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 23:08:45 (1790737725)
[ 4053.349579] Lustre: Mounted lustre-client - version 2.17.59_3_g4e55f3f
[ 4084.192196] Lustre: 2358:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790737742/real 1790737742]  req@ffff91b503e80000 x1877720440924032/t0(0) o400->lustre-MDT0000-mdc-ffff91b511253800@192.168.204.134@tcp:12/10 lens 224/224 e 0 to 1 dl 1790737758 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4084.231054] Lustre: 2358:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 4084.250690] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 192.168.204.134@tcp) was lost; in progress operations using this service will fail
[ 4084.287317] LustreError: Skipped 3 previous similar messages
[ 4094.454311] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0xb43ed9d3a2c672fc to 0xb43ed9d3a2da8ba8
[ 4094.466521] Lustre: Skipped 3 previous similar messages
[ 4115.620446] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4117.163083] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4119.384878] Lustre: Unmounted lustre-client
[ 4125.900268] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 23:09:58 (1790737798)
[ 4247.904287] Lustre: 2355:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790737905/real 1790737905]  req@ffff91b5059c8700 x1877720440959232/t0(0) o4->lustre-OST0000-osc-ffff91b511253800@192.168.204.134@tcp:6/4 lens 488/448 e 0 to 1 dl 1790737921 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4247.946806] Lustre: 2355:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 50 previous similar messages
[ 4423.085914] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 23:14:55 (1790738095)
[ 4475.034326] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 23:15:47 (1790738147)
[ 4479.235217] Lustre: Unmounted lustre-client
[ 4505.650954] Lustre: Mounted lustre-client - version 2.17.59_3_g4e55f3f
[ 4512.806424] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 23:16:25 (1790738185)
[ 4513.265073] LustreError: 56513:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4514.324808] LustreError: 56513:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4514.517124] Lustre: *** cfs_fail_loc=305, val=0***
[ 4539.304902] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 23:16:51 (1790738211)
[ 4540.077347] Lustre: Mounted lustre-client - version 2.17.59_3_g4e55f3f
[ 4543.212895] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4553.559550] Lustre: Unmounted lustre-client
[ 4560.790417] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 23:17:13 (1790738233)
[ 4664.092959] LustreError: lustre-MDT0000-mdc-ffff91b5057b0800: operation mds_reint to node 192.168.204.134@tcp failed: rc = -19
[ 4664.101122] LustreError: Skipped 1 previous similar message
[ 4664.108546] Lustre: lustre-MDT0000-mdc-ffff91b5057b0800: Connection to lustre-MDT0000 (at 192.168.204.134@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4664.130762] Lustre: Skipped 23 previous similar messages
[ 4685.793655] Lustre: 2357:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790738343/real 1790738343]  req@ffff91b50884ce00 x1877720443639552/t0(0) o400->MGC192.168.204.134@tcp@192.168.204.134@tcp:26/25 lens 224/224 e 0 to 1 dl 1790738359 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4685.843364] Lustre: 2357:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 69 previous similar messages
[ 4685.860285] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 192.168.204.134@tcp) was lost; in progress operations using this service will fail
[ 4695.092518] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0xb43ed9d3a2daa8f8 to 0xb43ed9d3a2dd5ec3
[ 4695.124210] Lustre: MGC192.168.204.134@tcp: Connection restored to 192.168.204.134@tcp (at 192.168.204.134@tcp)
[ 4695.140614] Lustre: Skipped 24 previous similar messages
[ 4868.790167] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 23:22:21 (1790738541)
[ 4873.766632] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4889.064677] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 192.168.204.134@tcp) was lost; in progress operations using this service will fail
[ 4889.093135] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0xb43ed9d3a2dd5ec3 to 0xb43ed9d3a2e227e6
[ 4889.137934] LustreError: lustre-MDT0000-mdc-ffff91b5057b0800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4902.400813] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 23:22:55 (1790738575)
[ 4903.040859] Lustre: Mounted lustre-client - version 2.17.59_3_g4e55f3f
[ 4910.510426] LustreError: lustre-OST0000-osc-ffff91b5057b0800: operation ldlm_enqueue to node 192.168.204.134@tcp failed: rc = -107
[ 4910.531581] LustreError: lustre-OST0000-osc-ffff91b5057b0800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4910.553382] LustreError: 60236:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0000-osc-ffff91b5057b0800: namespace resource [0x240000400:0x3da3:0x0].0x0 (ffff91b503ece500) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4923.314754] Lustre: Unmounted lustre-client
[ 4930.514731] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 23:23:23 (1790738603)
[ 4937.080063] LustreError: lustre-MDT0000-mdc-ffff91b5057b0800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4937.098852] LustreError: 60909:0:(file.c:6243:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4943.933210] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 23:23:36 (1790738616)
[ 4944.107512] LustreError: 61607:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4948.251945] LustreError: lustre-MDT0000-mdc-ffff91b5057b0800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4948.272735] LustreError: 61630:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4948.291134] LustreError: 61607:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=844
[ 4948.298355] LustreError: 61607:0:(import.c:762:ptlrpc_connect_import_locked()) already connecting
[ 4948.354848] LustreError: 61635:0:(file.c:6243:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4949.440654] LustreError: 61641:0:(file.c:6243:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4949.451138] LustreError: 61641:0:(file.c:6243:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4951.780054] LustreError: 61663:0:(file.c:6243:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4951.794951] LustreError: 61663:0:(file.c:6243:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4956.327086] LustreError: 61708:0:(file.c:6243:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4956.343732] LustreError: 61708:0:(file.c:6243:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4965.295517] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 23:23:58 (1790738638)
[ 5004.483382] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5006.233981] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5016.275769] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 23:24:48 (1790738688)
[ 5053.617833] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5055.012980] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5064.449726] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 23:25:37 (1790738737)
[ 5125.710969] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5127.198355] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5135.138247] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 23:26:47 (1790738807)
[ 5176.760144] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5178.283724] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5185.064621] Lustre: Unmounted lustre-client
[ 5237.365303] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5239.720967] Lustre: Mounted lustre-client - version 2.17.59_3_g4e55f3f
[ 5246.706053] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 23:28:39 (1790738919)
[ 5249.343496] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5250.949865] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 23:28:43 (1790738923)
[ 5284.210814] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 23:29:16 (1790738956)
[ 5285.551809] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5287.564796] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 23:29:19 (1790738959)
[ 5288.845987] Lustre: *** cfs_fail_loc=805, val=0***
[ 5288.896670] Lustre: Mounted lustre-client - version 2.17.59_3_g4e55f3f
[ 5294.341811] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5315.360295] Lustre: 2355:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790738973/real 1790738973]  req@ffff91b5086ec380 x1877720445691136/t0(0) o400->lustre-MDT0000-mdc-ffff91b518965000@192.168.204.134@tcp:12/10 lens 224/224 e 0 to 1 dl 1790738989 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5315.386832] Lustre: 2355:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5315.393559] Lustre: lustre-MDT0000-mdc-ffff91b518965000: Connection to lustre-MDT0000 (at 192.168.204.134@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5315.403153] Lustre: Skipped 12 previous similar messages
[ 5315.407372] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 192.168.204.134@tcp) was lost; in progress operations using this service will fail
[ 5325.804178] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0xb43ed9d3a2e231b0 to 0xb43ed9d3a2e2354c
[ 5325.830563] Lustre: lustre-MDT0000-mdc-ffff91b5055f0800: Connection restored to 192.168.204.134@tcp (at 192.168.204.134@tcp)
[ 5325.866242] Lustre: Skipped 13 previous similar messages
[ 5336.503721] Lustre: Unmounted lustre-client
[ 5343.949642] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 23:30:16 (1790739016)
[ 5371.922481] LustreError: 2354:0:(client.c:3449:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff91b508800000 x1877720445702912/t94489280516(94489280516) o101->lustre-MDT0000-mdc-ffff91b518965000@192.168.204.134@tcp:12/10 lens 664/608 e 0 to 0 dl 1790739061 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5379.767670] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5381.363158] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5390.821810] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 23:31:03 (1790739063)
[ 5392.552322] LustreError: lustre-OST0000-osc-ffff91b518965000: operation ost_write to node 192.168.204.134@tcp failed: rc = -107
[ 5392.576491] LustreError: Skipped 2 previous similar messages
[ 5392.593580] LustreError: lustre-OST0000-osc-ffff91b518965000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5392.618992] Lustre: 2357:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.134@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5392.690188] LustreError: 72814:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0000-osc-ffff91b518965000: namespace resource [0x240000400:0x3e02:0x0].0x0 (ffff91b503eceb00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5392.719715] LustreError: 72814:0:(ldlm_resource.c:1227:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5401.392365] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 23:31:13 (1790739073)
[ 5403.053845] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5405.040429] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 23:31:17 (1790739077)
[ 5406.580737] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5408.196710] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 23:31:20 (1790739080)
[ 5409.355974] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5411.145933] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 23:31:23 (1790739083)
[ 5412.486819] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5414.058458] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 23:31:26 (1790739086)
[ 5415.555131] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5417.072778] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 23:31:29 (1790739089)
[ 5418.638346] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5420.229836] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 23:31:32 (1790739092)
[ 5421.719858] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5423.147388] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 23:31:35 (1790739095)
[ 5424.547939] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5426.344062] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 23:31:38 (1790739098)
[ 5427.581600] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5429.596778] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 23:31:42 (1790739102)
[ 5431.216525] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5433.522441] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 23:31:45 (1790739105)
[ 5435.009640] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5436.857566] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 23:31:49 (1790739109)
[ 5439.210617] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5441.012367] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 23:31:53 (1790739113)
[ 5464.048520] Lustre: 68277:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.134@tcp: IR log lustre-cliir failed, not fatal: rc = -5
[ 5474.878650] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 23:32:27 (1790739147)
[ 5503.897333] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 23:32:56 (1790739176)
[ 5504.449864] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5517.493524] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 23:33:09 (1790739189)
[ 5518.325126] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5529.471518] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 23:33:21 (1790739201)
[ 5529.876893] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5537.894695] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 23:33:30 (1790739210)
[ 5538.487787] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5547.283355] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 23:33:39 (1790739219)
[ 5547.640060] Lustre: *** cfs_fail_loc=510, val=3***
[ 5556.684475] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 23:33:49 (1790739229)
[ 5557.085876] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5566.616054] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 23:33:59 (1790739239)
[ 5567.063418] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5632.394164] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 23:35:05 (1790739305)
[ 5632.806532] LustreError: 83031:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5635.598311] LustreError: lustre-MDT0000-mdc-ffff91b518965000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5635.626242] LustreError: 83045:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff91b518965000: inode [0x20000a042:0xf:0x0] mdc close failed: rc = -108
[ 5635.666913] LustreError: 83045:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff91b518965000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff91b503ecec00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5636.888810] LustreError: 83031:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5639.015961] LustreError: 83061:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5641.734229] LustreError: lustre-MDT0000-mdc-ffff91b518965000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5641.766394] LustreError: 83075:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff91b518965000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff91b503ecea00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5641.797881] LustreError: 83075:0:(ldlm_resource.c:1227:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5643.080147] LustreError: 83061:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5650.037361] LustreError: lustre-MDT0000-mdc-ffff91b518965000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5650.388275] LustreError: 83117:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5653.207489] LustreError: lustre-MDT0000-mdc-ffff91b518965000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5653.317537] LustreError: 83136:0:(file.c:6243:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5653.344094] LustreError: 83136:0:(file.c:6243:ll_inode_revalidate_fini()) Skipped 14 previous similar messages
[ 5654.476107] LustreError: 83117:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5654.482725] LustreError: 83117:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff91b518965000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5654.500706] LustreError: 83117:0:(file.c:252:ll_close_inode_openhandle()) Skipped 1 previous similar message
[ 5654.556443] LustreError: 83143:0:(file.c:6243:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5654.573499] LustreError: 83143:0:(file.c:6243:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5656.841652] LustreError: 83165:0:(file.c:6243:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5656.852412] LustreError: 83165:0:(file.c:6243:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5660.395498] LustreError: 83192:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5663.301931] LustreError: lustre-MDT0000-mdc-ffff91b518965000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5663.328762] LustreError: 83206:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff91b518965000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff91b503ed3300) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5663.344197] LustreError: 83206:0:(ldlm_resource.c:1227:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5664.472137] LustreError: 83192:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5671.375567] LustreError: lustre-MDT0000-mdc-ffff91b518965000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5671.416581] LustreError: 83235:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff91b518965000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff91b50930d000) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5671.446889] LustreError: 83235:0:(ldlm_resource.c:1227:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5671.808269] LustreError: 83248:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5674.592333] LustreError: lustre-MDT0000-mdc-ffff91b518965000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5675.880192] LustreError: 83248:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5679.036883] LustreError: lustre-MDT0000-mdc-ffff91b518965000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5679.148158] LustreError: 83295:0:(file.c:6243:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5679.157674] LustreError: 83295:0:(file.c:6243:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5680.113181] LustreError: 83276:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff91b518965000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5688.100644] LustreError: 83347:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5688.121048] LustreError: 83347:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5691.327772] LustreError: lustre-MDT0000-mdc-ffff91b518965000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5691.380926] LustreError: 83370:0:(file.c:6243:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5691.386329] LustreError: 83370:0:(file.c:6243:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5692.202877] LustreError: 83347:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5692.211769] LustreError: 83347:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5692.229353] LustreError: 83347:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff91b518965000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5701.887289] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 23:36:14 (1790739374)
[ 5722.020650] LustreError: 68274:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5722.026207] LustreError: 68274:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5726.096424] LustreError: 68274:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5726.105410] LustreError: 68274:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5732.762353] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 23:36:45 (1790739405)
[ 5783.817589] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 23:37:36 (1790739456)
[ 5850.268081] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 23:38:42 (1790739522)
[ 5885.452234] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 23:39:18 (1790739558)
[ 5886.004432] Lustre: Mounted lustre-client - version 2.17.59_3_g4e55f3f
[ 6009.970237] Lustre: Unmounted lustre-client
[ 6018.958577] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 23:41:31 (1790739691)
[ 6019.417395] LustreError: 87673:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 6022.631679] LustreError: lustre-OST0000-osc-ffff91b518965000: operation ost_statfs to node 192.168.204.134@tcp failed: rc = -107
[ 6022.646584] LustreError: Skipped 9 previous similar messages
[ 6022.658428] Lustre: lustre-OST0000-osc-ffff91b518965000: Connection to lustre-OST0000 (at 192.168.204.134@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6022.674446] Lustre: Skipped 17 previous similar messages
[ 6022.702118] LustreError: lustre-OST0000-osc-ffff91b518965000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6022.746893] LustreError: 87761:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0000-osc-ffff91b518965000: namespace resource [0x240000400:0x3e2d:0x0].0x0 (ffff91b50930dc00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6022.787492] LustreError: 87761:0:(ldlm_resource.c:1227:ldlm_resource_complain()) Skipped 2 previous similar messages
[ 6023.120216] LustreError: 87673:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout interrupted
[ 6023.128597] Lustre: 2356:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.134@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 6023.156062] Lustre: lustre-OST0000-osc-ffff91b518965000: Connection restored to 192.168.204.134@tcp (at 192.168.204.134@tcp)
[ 6023.173780] Lustre: Skipped 18 previous similar messages
[ 6030.322749] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 23:41:42 (1790739702)
[ 6087.648183] Lustre: 88340:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790739706/real 1790739706]  req@ffff91b508803800 x1877720445873280/t0(0) o101->lustre-MDT0000-mdc-ffff91b518965000@192.168.204.134@tcp:12/10 lens 328/344 e 0 to 1 dl 1790739761 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 6087.688820] Lustre: 88340:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 19 previous similar messages
[ 6096.114177] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 23:42:48 (1790739768)
[ 6098.177677] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 6100.231664] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 23:42:52 (1790739772)
[ 6164.758759] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6166.281836] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 23:43:58 (1790739838)
[ 6225.947980] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 23:44:58 (1790739898)
[ 6228.638932] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6230.387464] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 23:45:02 (1790739902)
[ 6232.088805] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6233.846548] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 23:45:06 (1790739906)
[ 6268.659747] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 23:45:41 (1790739941)
[ 6280.993412] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6306.789588] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 192.168.204.134@tcp) was lost; in progress operations using this service will fail
[ 6306.816979] LustreError: Skipped 3 previous similar messages
[ 6317.044965] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0xb43ed9d3a2e24012 to 0xb43ed9d3a2e259d4
[ 6317.056766] Lustre: Skipped 3 previous similar messages
[ 6332.096968] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6333.826629] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6345.604986] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 23:46:58 (1790740018)
[ 6347.817271] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6349.555056] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 23:47:02 (1790740022)
[ 6363.114766] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 192.168.204.134@tcp) was lost; in progress operations using this service will fail
[ 6363.147469] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0xb43ed9d3a2e259d4 to 0xb43ed9d3a2e25d07
[ 6377.157847] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 23:47:29 (1790740049)
[ 6429.107448] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 23:48:21 (1790740101)
[ 6498.916604] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6502.488842] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6586.274354] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 192.168.204.134@tcp) was lost; in progress operations using this service will fail
[ 6586.295392] LustreError: Skipped 1 previous similar message
[ 6596.661833] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0xb43ed9d3a2e26064 to 0xb43ed9d3a2e27047
[ 6596.683391] Lustre: Skipped 1 previous similar message
[ 6602.900488] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6604.344124] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6627.317159] Lustre: MGC192.168.204.134@tcp: Connection restored to 192.168.204.134@tcp (at 192.168.204.134@tcp)
[ 6627.328398] Lustre: Skipped 11 previous similar messages
[ 6628.657251] Lustre: 68277:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.134@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6637.560361] Lustre: lustre-MDT0000-mdc-ffff91b518965000: Connection to lustre-MDT0000 (at 192.168.204.134@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6637.581339] Lustre: Skipped 8 previous similar messages
[ 6649.389700] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6650.760335] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6687.712365] Lustre: 2358:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790740301/real 1790740301]  req@ffff91b528d44380 x1877720452406656/t0(0) o400->lustre-MDT0000-mdc-ffff91b518965000@192.168.204.134@tcp:12/10 lens 224/224 e 0 to 1 dl 1790740361 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6687.741316] Lustre: 2358:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 26 previous similar messages
[ 6794.644145] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 23:54:27 (1790740467)
[ 6805.837613] LustreError: lustre-OST0000-osc-ffff91b518965000: operation ost_setattr to node 192.168.204.134@tcp failed: rc = -19
[ 6805.851202] LustreError: Skipped 67 previous similar messages
[ 6858.361365] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6861.945216] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 7117.153842] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 23:59:49 (1790740789)
[ 7288.297861] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 192.168.204.134@tcp) was lost; in progress operations using this service will fail
[ 7288.309572] LustreError: Skipped 1 previous similar message
[ 7288.343760] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0xb43ed9d3a2e2708d to 0xb43ed9d3a2f1dbf3
[ 7288.353173] Lustre: Skipped 1 previous similar message
[ 7288.359822] Lustre: MGC192.168.204.134@tcp: Connection restored to 192.168.204.134@tcp (at 192.168.204.134@tcp)
[ 7288.367767] Lustre: Skipped 1 previous similar message
[ 7290.393800] Lustre: lustre-MDT0000-mdc-ffff91b518965000: Connection to lustre-MDT0000 (at 192.168.204.134@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7290.428716] Lustre: Skipped 1 previous similar message
[ 7315.425234] Lustre: 2357:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790740951/real 1790740952]  req@ffff91b50873a680 x1877720463348608/t0(0) o400->lustre-MDT0000-mdc-ffff91b518965000@192.168.204.134@tcp:12/10 lens 224/224 e 0 to 1 dl 1790740989 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7315.456663] Lustre: 2357:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 7325.281798] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 00:03:17 (1790740997)
[ 7327.104662] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7329.082621] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 00:03:21 (1790741001)
[ 7331.466332] LustreError: lustre-MDT0000-mdc-ffff91b518965000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7331.483943] LustreError: 102047:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff91b518965000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff91b509346b00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 7331.510432] LustreError: 102047:0:(ldlm_resource.c:1227:ldlm_resource_complain()) Skipped 1 previous similar message
[ 7339.532628] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 00:03:31 (1790741011)
[ 7506.405253] LustreError: lustre-OST0000-osc-ffff91b518965000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7508.776237] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 00:06:20 (1790741180)
[ 7553.621188] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 00:07:05 (1790741225)
[ 7555.689305] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7557.686632] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 00:07:09 (1790741229)
[ 7559.578804] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7561.665746] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 00:07:13 (1790741233)
[ 7596.993329] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 00:07:49 (1790741269)
[ 7636.463482] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 192.168.204.134@tcp) was lost; in progress operations using this service will fail
[ 7636.501466] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0xb43ed9d3a2f1dbf3 to 0xb43ed9d3a2f20b64
[ 7636.515742] Lustre: Skipped 1 previous similar message
[ 7654.090634] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 00:08:46 (1790741326)
[ 7655.852441] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7657.556393] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 00:08:50 (1790741330)
[ 7659.220604] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7661.028740] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 00:08:53 (1790741333)
[ 7673.837952] Lustre: Unmounted lustre-client
[ 7676.326291] Lustre: Mounted lustre-client - version 2.17.59_3_g4e55f3f
[ 7679.389448] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7712.467787] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2