[ 1885.520184] LustreError: 34731:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 1887.201327] LustreError: 8:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 1887.737396] LustreError: lustre-OST0000-osc-ffff932b87ecb000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1895.527173] LustreError: 34731:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 awake
[ 1895.532707] Lustre: 2356:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.121@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1895.656034] LustreError: 8:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout interrupted
[ 1895.661061] Lustre: 2355:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.121@tcp:/lustre/fid: [0x200000404:0x36:0x0]// may get corrupted (rc -108)
[ 1897.353642] Lustre: DEBUG MARKER: recovery-small test_24b: @@@@@@ IGNORE (bz5494): multiop didn't fail fsync: 108 or close: 0
[ 1906.122590] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 20:43:46 (1790729026)
[ 1908.228277] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1910.094922] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 20:43:49 (1790729029)
[ 1912.284957] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1914.217819] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 20:43:53 (1790729033)
[ 1918.477602] LustreError: lustre-MDT0000-mdc-ffff932b87ecb000: operation ldlm_enqueue to node 192.168.202.121@tcp failed: rc = -19
[ 1938.912292] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 192.168.202.121@tcp) was lost; in progress operations using this service will fail
[ 1938.946518] Lustre: Evicted from MGS (at 192.168.202.121@tcp) after server handle changed from 0xae41d90f10645e07 to 0xae41d90f10647783
[ 1940.929494] Lustre: 15901:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.121@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2050.670675] LustreError: lustre-MDT0000-mdc-ffff932b87ecb000: operation mds_reint to node 192.168.202.121@tcp failed: rc = -19
[ 2070.015180] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 192.168.202.121@tcp) was lost; in progress operations using this service will fail
[ 2070.077795] Lustre: Evicted from MGS (at 192.168.202.121@tcp) after server handle changed from 0xae41d90f10647783 to 0xae41d90f1066a4ab
[ 2084.101605] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 20:46:44 (1790729204)
[ 2084.645196] Lustre: *** cfs_fail_loc=305, val=0***
[ 2084.648782] Lustre: Skipped 2 previous similar messages
[ 2127.328987] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 192.168.202.121@tcp) was lost; in progress operations using this service will fail
[ 2137.592619] Lustre: Evicted from MGS (at 192.168.202.121@tcp) after server handle changed from 0xae41d90f1066a4ab to 0xae41d90f1066d614
[ 2151.537543] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2153.612090] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2162.408819] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 20:48:02 (1790729282)
[ 2178.536218] Lustre: lustre-MDT0000-mdc-ffff932b87ecb000: Connection to lustre-MDT0000 (at 192.168.202.121@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2178.549977] Lustre: Skipped 12 previous similar messages
[ 2178.563248] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 192.168.202.121@tcp) was lost; in progress operations using this service will fail
[ 2178.584282] Lustre: Evicted from MGS (at 192.168.202.121@tcp) after server handle changed from 0xae41d90f1066d614 to 0xae41d90f1066da6d
[ 2178.596061] Lustre: MGC192.168.202.121@tcp: Connection restored to 192.168.202.121@tcp (at 192.168.202.121@tcp)
[ 2178.603854] Lustre: Skipped 15 previous similar messages
[ 2178.637938] LustreError: lustre-MDT0000-mdc-ffff932b87ecb000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2198.049637] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 20:48:37 (1790729317)
[ 2212.818185] LustreError: lustre-OST0000-osc-ffff932b87ecb000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2237.396516] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 20:49:17 (1790729357)
[ 2250.784935] LustreError: lustre-MDT0000-mdc-ffff932b87ecb000: operation mds_reint to node 192.168.202.121@tcp failed: rc = -19
[ 2250.798090] LustreError: Skipped 1 previous similar message
[ 2270.687971] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 192.168.202.121@tcp) was lost; in progress operations using this service will fail
[ 2270.750631] Lustre: Evicted from MGS (at 192.168.202.121@tcp) after server handle changed from 0xae41d90f1066da6d to 0xae41d90f10672981
[ 2272.827780] Lustre: 15901:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.121@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2301.102300] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2303.932282] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2388.448245] Lustre: 2354:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790729493/real 1790729493]  req@ffff932b831da300 x1877713563229312/t0(0) o400->MGC192.168.202.121@tcp@192.168.202.121@tcp:26/25 lens 224/224 e 0 to 1 dl 1790729509 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2388.516064] Lustre: 2354:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 14 previous similar messages
[ 2388.536420] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 192.168.202.121@tcp) was lost; in progress operations using this service will fail
[ 2397.752963] Lustre: Evicted from MGS (at 192.168.202.121@tcp) after server handle changed from 0xae41d90f10672981 to 0xae41d90f1068a83c
[ 2417.573129] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2419.912872] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2485.857385] LustreError: lustre-MDT0000-mdc-ffff932b87ecb000: operation mds_close to node 192.168.202.121@tcp failed: rc = -19
[ 2485.880557] LustreError: Skipped 3 previous similar messages
[ 2505.248660] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 192.168.202.121@tcp) was lost; in progress operations using this service will fail
[ 2515.452630] Lustre: Evicted from MGS (at 192.168.202.121@tcp) after server handle changed from 0xae41d90f1068a83c to 0xae41d90f106a7953
[ 2533.260313] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2536.355670] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2567.184596] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 20:54:46 (1790729686)
[ 2593.247772] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 192.168.202.121@tcp) was lost; in progress operations using this service will fail
[ 2603.442365] Lustre: Evicted from MGS (at 192.168.202.121@tcp) after server handle changed from 0xae41d90f106a7953 to 0xae41d90f106b443e
[ 2615.718813] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2657.254741] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2695.047073] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2728.415615] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 192.168.202.121@tcp) was lost; in progress operations using this service will fail
[ 2728.426561] LustreError: Skipped 2 previous similar messages
[ 2738.693452] Lustre: Evicted from MGS (at 192.168.202.121@tcp) after server handle changed from 0xae41d90f106b9d31 to 0xae41d90f106bcd12
[ 2738.714896] Lustre: Skipped 2 previous similar messages
[ 2743.825291] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2767.348114] LustreError: lustre-MDT0000-mdc-ffff932b87ecb000: operation mds_close to node 192.168.202.121@tcp failed: rc = -19
[ 2767.354250] LustreError: Skipped 8 previous similar messages
[ 2789.121701] Lustre: MGC192.168.202.121@tcp: Connection restored to 192.168.202.121@tcp (at 192.168.202.121@tcp)
[ 2789.135459] Lustre: Skipped 16 previous similar messages
[ 2795.782091] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2824.535151] Lustre: lustre-MDT0000-mdc-ffff932b87ecb000: Connection to lustre-MDT0000 (at 192.168.202.121@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2824.551543] Lustre: Skipped 9 previous similar messages
[ 2854.670389] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2950.299941] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 21:01:10 (1790730070)
[ 3001.516909] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3003.681726] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3298.677733] LustreError: lustre-OST0000-osc-ffff932b87ecb000: operation ldlm_enqueue to node 192.168.202.121@tcp failed: rc = -19
[ 3298.701808] LustreError: Skipped 4 previous similar messages
[ 3343.645295] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3346.110541] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3630.162898] Lustre: lustre-OST0000-osc-ffff932b87ecb000: Connection to lustre-OST0000 (at 192.168.202.121@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3630.182774] Lustre: Skipped 3 previous similar messages
[ 3653.187033] Lustre: lustre-OST0000-osc-ffff932b87ecb000: Connection restored to 192.168.202.121@tcp (at 192.168.202.121@tcp)
[ 3653.196719] Lustre: Skipped 7 previous similar messages
[ 3674.658346] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3676.808604] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3926.602470] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 21:17:26 (1790731046)
[ 3943.903789] Lustre: 50833:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790731049/real 1790731049]  req@ffff932b85a93100 x1877713586722432/t0(0) o101->lustre-MDT0000-mdc-ffff932b87ecb000@192.168.202.121@tcp:12/10 lens 576/1152 e 0 to 1 dl 1790731065 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3943.951225] Lustre: 50833:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
[ 3953.198857] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 21:17:53 (1790731073)
[ 3979.313600] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 21:18:19 (1790731099)
[ 4007.638859] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 21:18:47 (1790731127)
[ 4008.212799] Lustre: Mounted lustre-client - version 2.17.59_2_g85450b5
[ 4039.967159] Lustre: 2353:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790731145/real 1790731145]  req@ffff932b8509aa00 x1877713586742144/t0(0) o400->lustre-MDT0000-mdc-ffff932b87ecb000@192.168.202.121@tcp:12/10 lens 224/224 e 0 to 1 dl 1790731161 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4039.998964] Lustre: 2353:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 4040.030111] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 192.168.202.121@tcp) was lost; in progress operations using this service will fail
[ 4040.038016] LustreError: Skipped 3 previous similar messages
[ 4040.048083] Lustre: Evicted from MGS (at 192.168.202.121@tcp) after server handle changed from 0xae41d90f106dcf06 to 0xae41d90f10843e08
[ 4040.064953] Lustre: Skipped 3 previous similar messages
[ 4053.990365] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4055.841377] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4057.731649] Lustre: Unmounted lustre-client
[ 4064.399977] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 21:19:44 (1790731184)
[ 4201.887666] Lustre: 2354:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790731307/real 1790731307]  req@ffff932b8509b100 x1877713586776192/t0(0) o4->lustre-OST0000-osc-ffff932b87ecb000@192.168.202.121@tcp:6/4 lens 488/448 e 0 to 1 dl 1790731323 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4201.912981] Lustre: 2354:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 56 previous similar messages
[ 4233.695394] Lustre: lustre-OST0000-osc-ffff932b87ecb000: Connection to lustre-OST0000 (at 192.168.202.121@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4233.719959] Lustre: Skipped 14 previous similar messages
[ 4266.489859] Lustre: lustre-OST0000-osc-ffff932b87ecb000: Connection restored to 192.168.202.121@tcp (at 192.168.202.121@tcp)
[ 4266.511813] Lustre: Skipped 17 previous similar messages
[ 4362.348992] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 21:24:41 (1790731481)
[ 4412.790849] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 21:25:32 (1790731532)
[ 4416.749760] Lustre: Unmounted lustre-client
[ 4442.053299] Lustre: Mounted lustre-client - version 2.17.59_2_g85450b5
[ 4450.292234] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 21:26:09 (1790731569)
[ 4450.633303] LustreError: 56759:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4451.702625] LustreError: 56759:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4451.991172] Lustre: *** cfs_fail_loc=305, val=0***
[ 4475.218994] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 21:26:34 (1790731594)
[ 4475.730572] Lustre: Mounted lustre-client - version 2.17.59_2_g85450b5
[ 4477.822302] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4488.181272] Lustre: Unmounted lustre-client
[ 4497.799052] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 21:26:57 (1790731617)
[ 4585.872343] LustreError: lustre-MDT0000-mdc-ffff932b831d7800: operation mds_reint to node 192.168.202.121@tcp failed: rc = -19
[ 4585.883933] LustreError: Skipped 1 previous similar message
[ 4604.895491] Lustre: 2353:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790731710/real 1790731710]  req@ffff932b87b0ea00 x1877713589448704/t0(0) o400->MGC192.168.202.121@tcp@192.168.202.121@tcp:26/25 lens 224/224 e 0 to 1 dl 1790731726 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4604.918664] Lustre: 2353:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 87 previous similar messages
[ 4604.928832] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 192.168.202.121@tcp) was lost; in progress operations using this service will fail
[ 4615.158788] Lustre: Evicted from MGS (at 192.168.202.121@tcp) after server handle changed from 0xae41d90f1084574c to 0xae41d90f10870ad2
[ 4778.051305] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 21:31:38 (1790731898)
[ 4784.222804] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4798.446979] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 192.168.202.121@tcp) was lost; in progress operations using this service will fail
[ 4798.470802] Lustre: Evicted from MGS (at 192.168.202.121@tcp) after server handle changed from 0xae41d90f10870ad2 to 0xae41d90f108bd839
[ 4808.718951] LustreError: lustre-MDT0000-mdc-ffff932b831d7800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4820.213316] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 21:32:19 (1790731939)
[ 4820.811161] Lustre: Mounted lustre-client - version 2.17.59_2_g85450b5
[ 4828.393354] LustreError: lustre-OST0000-osc-ffff932b831d7800: operation ldlm_enqueue to node 192.168.202.121@tcp failed: rc = -107
[ 4828.413219] LustreError: lustre-OST0000-osc-ffff932b831d7800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4838.368316] Lustre: lustre-OST0000-osc-ffff932b8a521000: Connection to lustre-OST0000 (at 192.168.202.121@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4838.399291] Lustre: Skipped 12 previous similar messages
[ 4840.305970] Lustre: Unmounted lustre-client
[ 4847.639583] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 21:32:47 (1790731967)
[ 4854.438785] LustreError: lustre-MDT0000-mdc-ffff932b831d7800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4854.452776] LustreError: 61153:0:(file.c:6243:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4863.581505] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 21:33:03 (1790731983)
[ 4863.770912] LustreError: 61853:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4867.980708] LustreError: lustre-MDT0000-mdc-ffff932b831d7800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4867.993071] LustreError: 61875:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4868.027388] LustreError: 61853:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=772
[ 4868.049636] LustreError: 61853:0:(import.c:762:ptlrpc_connect_import_locked()) already connecting
[ 4868.064062] LustreError: 61880:0:(file.c:6243:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4869.169926] LustreError: 61886:0:(file.c:6243:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4869.192882] LustreError: 61886:0:(file.c:6243:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4871.579364] LustreError: 61908:0:(file.c:6243:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4871.597255] LustreError: 61908:0:(file.c:6243:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4876.474254] LustreError: 61953:0:(file.c:6243:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4876.512030] LustreError: 61953:0:(file.c:6243:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4878.307379] Lustre: lustre-MDT0000-mdc-ffff932b831d7800: Connection restored to 192.168.202.121@tcp (at 192.168.202.121@tcp)
[ 4878.314711] Lustre: Skipped 15 previous similar messages
[ 4886.566649] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 21:33:26 (1790732006)
[ 4927.570360] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4928.998258] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4941.576391] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 21:34:21 (1790732061)
[ 4982.190470] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4983.910568] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4994.945571] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 21:35:14 (1790732114)
[ 5059.239516] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5060.611294] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5069.070337] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 21:36:29 (1790732189)
[ 5115.515695] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5117.121582] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5124.398645] Lustre: Unmounted lustre-client
[ 5180.222443] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5182.796242] Lustre: Mounted lustre-client - version 2.17.59_2_g85450b5
[ 5190.388186] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 21:38:30 (1790732310)
[ 5192.856186] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5194.513982] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 21:38:34 (1790732314)
[ 5224.841365] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 21:39:04 (1790732344)
[ 5226.139174] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5227.869736] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 21:39:07 (1790732347)
[ 5229.178323] Lustre: *** cfs_fail_loc=805, val=0***
[ 5229.217675] Lustre: Mounted lustre-client - version 2.17.59_2_g85450b5
[ 5234.711396] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5256.160980] Lustre: 2353:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790732361/real 1790732361]  req@ffff932b85bbe300 x1877713591508352/t0(0) o400->lustre-MDT0000-mdc-ffff932b8be4d800@192.168.202.121@tcp:12/10 lens 224/224 e 0 to 1 dl 1790732377 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5256.182505] Lustre: 2353:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5256.188321] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 192.168.202.121@tcp) was lost; in progress operations using this service will fail
[ 5256.202121] Lustre: Evicted from MGS (at 192.168.202.121@tcp) after server handle changed from 0xae41d90f108be2ce to 0xae41d90f108be4e9
[ 5257.832403] Lustre: 68534:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.121@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 5273.367507] Lustre: Unmounted lustre-client
[ 5281.614677] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 21:40:01 (1790732401)
[ 5309.480600] LustreError: 2352:0:(client.c:3449:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff932b844ed880 x1877713591520768/t94489280516(94489280516) o101->lustre-MDT0000-mdc-ffff932b8be4d800@192.168.202.121@tcp:12/10 lens 664/608 e 0 to 0 dl 1790732446 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5319.674731] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5321.464155] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5329.213813] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 21:40:49 (1790732449)
[ 5330.492994] LustreError: lustre-OST0000-osc-ffff932b8be4d800: operation ost_write to node 192.168.202.121@tcp failed: rc = -107
[ 5330.506842] LustreError: Skipped 2 previous similar messages
[ 5330.525816] LustreError: lustre-OST0000-osc-ffff932b8be4d800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5330.548406] Lustre: 2356:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.121@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5330.607058] LustreError: 73074:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0000-osc-ffff932b8be4d800: namespace resource [0x240000400:0x4202:0x0].0x0 (ffff932b882bb100) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5338.119472] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 21:40:58 (1790732458)
[ 5339.779529] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5341.381805] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 21:41:01 (1790732461)
[ 5343.349464] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5345.457611] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 21:41:05 (1790732465)
[ 5347.181033] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5348.705358] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 21:41:08 (1790732468)
[ 5350.529663] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5352.391329] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 21:41:12 (1790732472)
[ 5353.778207] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5355.864122] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 21:41:15 (1790732475)
[ 5357.373800] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5358.935213] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 21:41:19 (1790732479)
[ 5360.470523] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5362.378914] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 21:41:22 (1790732482)
[ 5363.886869] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5365.995914] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 21:41:25 (1790732485)
[ 5367.540862] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5369.378296] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 21:41:29 (1790732489)
[ 5371.120384] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5372.988255] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 21:41:32 (1790732492)
[ 5375.300689] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5377.547242] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 21:41:37 (1790732497)
[ 5380.709490] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5382.689404] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 21:41:42 (1790732502)
[ 5406.193048] Lustre: 68534:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.121@tcp: IR log lustre-cliir failed, not fatal: rc = -5
[ 5420.658705] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 21:42:20 (1790732540)
[ 5451.930326] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 21:42:51 (1790732571)
[ 5452.450242] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5463.207475] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 21:43:03 (1790732583)
[ 5463.732346] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5474.828369] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 21:43:14 (1790732594)
[ 5475.164561] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5484.567723] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 21:43:24 (1790732604)
[ 5484.814568] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5493.727765] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 21:43:33 (1790732613)
[ 5494.246411] Lustre: *** cfs_fail_loc=510, val=3***
[ 5503.381440] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 21:43:43 (1790732623)
[ 5503.697854] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5563.871347] Lustre: lustre-OST0001-osc-ffff932b8be4d800: Connection to lustre-OST0001 (at 192.168.202.121@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5563.892028] Lustre: Skipped 14 previous similar messages
[ 5571.488730] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 21:44:51 (1790732691)
[ 5571.933994] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5642.277512] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 21:46:01 (1790732761)
[ 5642.864322] LustreError: 83292:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5645.687925] LustreError: lustre-MDT0000-mdc-ffff932b8be4d800: operation ldlm_enqueue to node 192.168.202.121@tcp failed: rc = -107
[ 5645.719385] LustreError: lustre-MDT0000-mdc-ffff932b8be4d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5645.755959] LustreError: 83307:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff932b8be4d800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff932b87b02300) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5645.808988] Lustre: lustre-MDT0000-mdc-ffff932b8be4d800: Connection restored to 192.168.202.121@tcp (at 192.168.202.121@tcp)
[ 5645.828490] Lustre: Skipped 12 previous similar messages
[ 5646.967209] LustreError: 83292:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5649.073766] LustreError: 83322:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5652.180096] LustreError: lustre-MDT0000-mdc-ffff932b8be4d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5652.211842] LustreError: 83337:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff932b8be4d800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff932b85b4e800) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5652.258852] LustreError: 83337:0:(ldlm_resource.c:1227:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5653.159135] LustreError: 83322:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5660.209552] LustreError: lustre-MDT0000-mdc-ffff932b8be4d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5660.575205] LustreError: 83379:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5663.768555] LustreError: lustre-MDT0000-mdc-ffff932b8be4d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5663.855967] LustreError: 83398:0:(file.c:6243:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5663.866762] LustreError: 83398:0:(file.c:6243:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5664.663618] LustreError: 83379:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5664.673342] LustreError: 83379:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff932b8be4d800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5665.016629] LustreError: 83404:0:(file.c:6243:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5665.028925] LustreError: 83404:0:(file.c:6243:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5667.423248] LustreError: 83427:0:(file.c:6243:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5667.435719] LustreError: 83427:0:(file.c:6243:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5671.245075] LustreError: 83454:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5674.438896] LustreError: lustre-MDT0000-mdc-ffff932b8be4d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5674.463393] LustreError: 83468:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff932b8be4d800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff932b882b2b00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5674.497807] LustreError: 83468:0:(ldlm_resource.c:1227:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5675.327279] LustreError: 83454:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5682.579989] LustreError: lustre-MDT0000-mdc-ffff932b8be4d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5682.588714] LustreError: 83497:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff932b8be4d800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff932b87b02400) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5682.600503] LustreError: 83497:0:(ldlm_resource.c:1227:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5683.033622] LustreError: 83510:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5686.066874] LustreError: lustre-MDT0000-mdc-ffff932b8be4d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5687.103194] LustreError: 83510:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5690.217894] LustreError: lustre-MDT0000-mdc-ffff932b8be4d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5690.293292] LustreError: 83557:0:(file.c:6243:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5690.300146] LustreError: 83557:0:(file.c:6243:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5691.304304] LustreError: 83538:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff932b8be4d800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5699.386701] LustreError: 83609:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5699.398715] LustreError: 83609:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5703.129196] LustreError: lustre-MDT0000-mdc-ffff932b8be4d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5703.247989] LustreError: 83632:0:(file.c:6243:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5703.253440] LustreError: 83632:0:(file.c:6243:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5703.407144] LustreError: 83609:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5703.416481] LustreError: 83609:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5703.432178] LustreError: 83609:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff932b8be4d800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5715.435132] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 21:47:15 (1790732835)
[ 5735.138823] LustreError: 68532:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5735.144832] LustreError: 68532:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5739.223191] LustreError: 68532:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5739.229794] LustreError: 68532:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5745.225183] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 21:47:45 (1790732865)
[ 5797.138887] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 21:48:37 (1790732917)
[ 5862.986350] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 21:49:42 (1790732982)
[ 5896.970204] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 21:50:17 (1790733017)
[ 5897.592759] Lustre: Mounted lustre-client - version 2.17.59_2_g85450b5
[ 6021.630316] Lustre: Unmounted lustre-client
[ 6028.473355] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 21:52:28 (1790733148)
[ 6029.088525] LustreError: 87937:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 6031.895094] LustreError: lustre-OST0000-osc-ffff932b8be4d800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6031.914957] LustreError: 88024:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0000-osc-ffff932b8be4d800: namespace resource [0x240000400:0x422d:0x0].0x0 (ffff932b87b02700) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6031.935391] LustreError: 88024:0:(ldlm_resource.c:1227:ldlm_resource_complain()) Skipped 2 previous similar messages
[ 6032.159109] LustreError: 87937:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout interrupted
[ 6032.162370] Lustre: 2353:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.121@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 6038.805852] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 21:52:38 (1790733158)
[ 6097.887523] Lustre: 88607:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790733163/real 1790733163]  req@ffff932b844ed880 x1877713591695488/t0(0) o101->lustre-MDT0000-mdc-ffff932b8be4d800@192.168.202.121@tcp:12/10 lens 328/344 e 0 to 1 dl 1790733219 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 6097.915652] Lustre: 88607:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 20 previous similar messages
[ 6105.970258] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 21:53:45 (1790733225)
[ 6107.540324] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 6109.682276] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 21:53:49 (1790733229)
[ 6167.519283] Lustre: lustre-MDT0000-mdc-ffff932b8be4d800: Connection to lustre-MDT0000 (at 192.168.202.121@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6167.551092] Lustre: Skipped 15 previous similar messages
[ 6174.702922] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6176.834057] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 21:54:56 (1790733296)
[ 6236.512842] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 21:55:55 (1790733355)
[ 6238.968168] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6240.730484] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 21:56:00 (1790733360)
[ 6242.596892] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6244.279678] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 21:56:04 (1790733364)
[ 6280.921334] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 21:56:40 (1790733400)
[ 6293.222753] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6322.143357] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 192.168.202.121@tcp) was lost; in progress operations using this service will fail
[ 6322.154267] LustreError: Skipped 3 previous similar messages
[ 6332.410918] Lustre: Evicted from MGS (at 192.168.202.121@tcp) after server handle changed from 0xae41d90f108bf130 to 0xae41d90f108c09ef
[ 6332.425091] Lustre: Skipped 3 previous similar messages
[ 6332.435281] Lustre: MGC192.168.202.121@tcp: Connection restored to 192.168.202.121@tcp (at 192.168.202.121@tcp)
[ 6332.458585] Lustre: Skipped 15 previous similar messages
[ 6349.017652] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6350.947165] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6362.972362] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 21:58:02 (1790733482)
[ 6367.051103] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6369.633432] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 21:58:08 (1790733488)
[ 6383.593159] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 192.168.202.121@tcp) was lost; in progress operations using this service will fail
[ 6383.631258] Lustre: Evicted from MGS (at 192.168.202.121@tcp) after server handle changed from 0xae41d90f108c09ef to 0xae41d90f108c0e95
[ 6399.201598] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 21:58:39 (1790733519)
[ 6448.244488] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 21:59:28 (1790733568)
[ 6456.019668] LustreError: lustre-OST0000-osc-ffff932b8be4d800: operation ost_setattr to node 192.168.202.121@tcp failed: rc = -107
[ 6456.045523] LustreError: Skipped 10 previous similar messages
[ 6509.718475] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6513.152549] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6599.583325] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 192.168.202.121@tcp) was lost; in progress operations using this service will fail
[ 6599.594232] LustreError: Skipped 1 previous similar message
[ 6599.616206] Lustre: Evicted from MGS (at 192.168.202.121@tcp) after server handle changed from 0xae41d90f108c10a2 to 0xae41d90f108c1feb
[ 6599.628779] Lustre: Skipped 1 previous similar message
[ 6615.678067] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6617.602681] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6667.903395] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6669.508821] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6701.024614] Lustre: 2356:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790733767/real 1790733767]  req@ffff932b9daf9180 x1877713598222336/t0(0) o400->lustre-MDT0000-mdc-ffff932b8be4d800@192.168.202.121@tcp:12/10 lens 224/224 e 0 to 1 dl 1790733822 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6701.085187] Lustre: 2356:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 30 previous similar messages
[ 6814.042454] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 22:05:33 (1790733933)
[ 6825.417029] Lustre: lustre-OST0000-osc-ffff932b8be4d800: Connection to lustre-OST0000 (at 192.168.202.121@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6825.433731] Lustre: Skipped 7 previous similar messages
[ 6885.426354] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6889.567204] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 7148.050577] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 22:11:08 (1790734268)
[ 7293.932136] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 192.168.202.121@tcp) was lost; in progress operations using this service will fail
[ 7293.952098] LustreError: Skipped 1 previous similar message
[ 7293.974248] Lustre: Evicted from MGS (at 192.168.202.121@tcp) after server handle changed from 0xae41d90f108c2309 to 0xae41d90f109bc6f4
[ 7293.979572] Lustre: Skipped 1 previous similar message
[ 7293.983318] Lustre: MGC192.168.202.121@tcp: Connection restored to 192.168.202.121@tcp (at 192.168.202.121@tcp)
[ 7294.001217] Lustre: Skipped 9 previous similar messages
[ 7323.281539] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 22:14:03 (1790734443)
[ 7324.639418] Lustre: 2355:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790734405/real 1790734405]  req@ffff932b8adb1c00 x1877713609379072/t0(0) o400->lustre-MDT0000-mdc-ffff932b8be4d800@192.168.202.121@tcp:12/10 lens 224/224 e 0 to 1 dl 1790734445 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7324.693163] Lustre: 2355:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 7325.170447] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7326.985735] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 22:14:06 (1790734446)
[ 7328.708061] LustreError: lustre-MDT0000-mdc-ffff932b8be4d800: operation ldlm_enqueue to node 192.168.202.121@tcp failed: rc = -107
[ 7328.718503] LustreError: Skipped 15 previous similar messages
[ 7328.726401] LustreError: lustre-MDT0000-mdc-ffff932b8be4d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7336.752784] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 22:14:16 (1790734456)
[ 7503.780844] LustreError: lustre-OST0000-osc-ffff932b8be4d800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7505.858732] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 22:17:05 (1790734625)
[ 7531.489916] Lustre: lustre-OST0000-osc-ffff932b8be4d800: Connection to lustre-OST0000 (at 192.168.202.121@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7531.510544] Lustre: Skipped 3 previous similar messages
[ 7550.270864] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 22:17:49 (1790734669)
[ 7551.974794] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7553.887309] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 22:17:53 (1790734673)
[ 7555.332157] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7556.814783] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 22:17:56 (1790734676)
[ 7590.504885] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 22:18:30 (1790734710)
[ 7631.856097] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 192.168.202.121@tcp) was lost; in progress operations using this service will fail
[ 7631.895652] Lustre: Evicted from MGS (at 192.168.202.121@tcp) after server handle changed from 0xae41d90f109bc6f4 to 0xae41d90f109bee85
[ 7631.903289] Lustre: Skipped 1 previous similar message
[ 7645.442890] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 22:19:25 (1790734765)
[ 7646.857757] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7648.678622] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 22:19:28 (1790734768)
[ 7650.429275] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7652.300508] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 22:19:32 (1790734772)
[ 7663.763917] Lustre: Unmounted lustre-client
[ 7666.292071] Lustre: Mounted lustre-client - version 2.17.59_2_g85450b5
[ 7670.119034] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7693.950369] Lustre: 107082:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.121@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 7712.437110] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2