[ 2025.373422] LustreError: lustre-OST0000-osc-ffff9dbb866ee800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2025.407131] Lustre: 2366:0:(llite_lib.c:4187:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.206.119@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 2032.787728] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 11:19:01 (1779203941)
[ 2035.148654] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 2037.382017] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 11:19:05 (1779203945)
[ 2039.664653] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 2042.755419] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 11:19:10 (1779203950)
[ 2066.399265] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 192.168.206.119@tcp) was lost; in progress operations using this service will fail
[ 2076.596455] Lustre: Evicted from MGS (at 192.168.206.119@tcp) after server handle changed from 0xda6eebafea6d58af to 0xda6eebafea6d7319
[ 2180.125803] LustreError: lustre-MDT0000-mdc-ffff9dbb866ee800: operation ldlm_enqueue to node 192.168.206.119@tcp failed: rc = -19
[ 2180.150209] LustreError: Skipped 4 previous similar messages
[ 2198.501857] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 192.168.206.119@tcp) was lost; in progress operations using this service will fail
[ 2208.689624] Lustre: Evicted from MGS (at 192.168.206.119@tcp) after server handle changed from 0xda6eebafea6d7319 to 0xda6eebafea6f76cd
[ 2223.141532] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 11:22:12 (1779204132)
[ 2223.585300] Lustre: *** cfs_fail_loc=305, val=0***
[ 2223.590610] Lustre: Skipped 2 previous similar messages
[ 2263.011449] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 192.168.206.119@tcp) was lost; in progress operations using this service will fail
[ 2273.268257] Lustre: lustre-MDT0000-mdc-ffff9dbb866ee800: Connection to lustre-MDT0000 (at 192.168.206.119@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2273.273269] Lustre: Evicted from MGS (at 192.168.206.119@tcp) after server handle changed from 0xda6eebafea6f76cd to 0xda6eebafea6f7a38
[ 2273.281771] Lustre: Skipped 11 previous similar messages
[ 2273.311765] Lustre: MGC192.168.206.119@tcp: Connection restored to 192.168.206.119@tcp (at 192.168.206.119@tcp)
[ 2273.325427] Lustre: Skipped 13 previous similar messages
[ 2290.847794] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2293.513748] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2304.240852] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 11:23:32 (1779204212)
[ 2319.352101] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 192.168.206.119@tcp) was lost; in progress operations using this service will fail
[ 2319.407033] Lustre: Evicted from MGS (at 192.168.206.119@tcp) after server handle changed from 0xda6eebafea6f7a38 to 0xda6eebafea6f7f08
[ 2319.564966] LustreError: lustre-MDT0000-mdc-ffff9dbb866ee800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2352.543733] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 11:24:20 (1779204260)
[ 2370.562798] LustreError: lustre-OST0000-osc-ffff9dbb866ee800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2389.035412] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 11:24:57 (1779204297)
[ 2402.841739] LustreError: lustre-MDT0000-mdc-ffff9dbb866ee800: operation mds_close to node 192.168.206.119@tcp failed: rc = -19
[ 2402.846969] LustreError: Skipped 1 previous similar message
[ 2422.752707] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 192.168.206.119@tcp) was lost; in progress operations using this service will fail
[ 2431.920700] Lustre: Evicted from MGS (at 192.168.206.119@tcp) after server handle changed from 0xda6eebafea6f7f08 to 0xda6eebafea6fcae9
[ 2447.198825] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2450.097687] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2533.919408] Lustre: 2367:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779204428/real 1779204428]  req@ffff9dbb84e9f480 x1865628544346240/t0(0) o400->MGC192.168.206.119@tcp@192.168.206.119@tcp:26/25 lens 224/224 e 0 to 1 dl 1779204444 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2533.980332] Lustre: 2367:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 13 previous similar messages
[ 2533.997457] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 192.168.206.119@tcp) was lost; in progress operations using this service will fail
[ 2544.260298] Lustre: Evicted from MGS (at 192.168.206.119@tcp) after server handle changed from 0xda6eebafea6fcae9 to 0xda6eebafea7172af
[ 2560.513740] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2562.684898] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2630.192223] LustreError: lustre-MDT0000-mdc-ffff9dbb866ee800: operation mds_reint to node 192.168.206.119@tcp failed: rc = -19
[ 2630.204627] LustreError: Skipped 1 previous similar message
[ 2650.100064] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 192.168.206.119@tcp) was lost; in progress operations using this service will fail
[ 2650.152940] Lustre: Evicted from MGS (at 192.168.206.119@tcp) after server handle changed from 0xda6eebafea7172af to 0xda6eebafea731ae5
[ 2651.241159] Lustre: 15918:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.206.119@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2676.074297] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2678.843457] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2709.008569] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 11:30:16 (1779204616)
[ 2732.575298] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 192.168.206.119@tcp) was lost; in progress operations using this service will fail
[ 2742.761848] Lustre: Evicted from MGS (at 192.168.206.119@tcp) after server handle changed from 0xda6eebafea731ae5 to 0xda6eebafea73e115
[ 2744.610758] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2790.311681] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2830.676225] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2865.632375] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 192.168.206.119@tcp) was lost; in progress operations using this service will fail
[ 2865.645616] LustreError: Skipped 2 previous similar messages
[ 2866.812547] Lustre: 15918:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.206.119@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2880.839396] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2884.189392] Lustre: lustre-MDT0000-mdc-ffff9dbb866ee800: Connection restored to 192.168.206.119@tcp (at 192.168.206.119@tcp)
[ 2884.214447] Lustre: Skipped 16 previous similar messages
[ 2904.751145] LustreError: lustre-MDT0000-mdc-ffff9dbb866ee800: operation ldlm_enqueue to node 192.168.206.119@tcp failed: rc = -19
[ 2904.756244] LustreError: Skipped 4 previous similar messages
[ 2904.758693] Lustre: lustre-MDT0000-mdc-ffff9dbb866ee800: Connection to lustre-MDT0000 (at 192.168.206.119@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2904.767392] Lustre: Skipped 8 previous similar messages
[ 2925.607534] Lustre: Evicted from MGS (at 192.168.206.119@tcp) after server handle changed from 0xda6eebafea74273f to 0xda6eebafea749501
[ 2925.622456] Lustre: Skipped 3 previous similar messages
[ 2926.963676] Lustre: 15918:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.206.119@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2941.539328] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 3009.308563] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 3109.189831] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 11:36:57 (1779205017)
[ 3165.647399] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3168.212146] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3455.261474] LustreError: lustre-OST0000-osc-ffff9dbb866ee800: operation ldlm_enqueue to node 192.168.206.119@tcp failed: rc = -107
[ 3455.278124] LustreError: Skipped 4 previous similar messages
[ 3497.652104] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3500.641781] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3786.177860] Lustre: lustre-OST0000-osc-ffff9dbb866ee800: Connection to lustre-OST0000 (at 192.168.206.119@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3786.203735] Lustre: Skipped 4 previous similar messages
[ 3811.300633] Lustre: lustre-OST0000-osc-ffff9dbb866ee800: Connection restored to 192.168.206.119@tcp (at 192.168.206.119@tcp)
[ 3811.308200] Lustre: Skipped 8 previous similar messages
[ 3830.749545] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3833.792416] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4081.350671] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 11:53:10 (1779205990)
[ 4099.042096] Lustre: 50261:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779205993/real 1779205993]  req@ffff9dbb85ad2680 x1865628564069504/t0(0) o101->lustre-MDT0000-mdc-ffff9dbb866ee800@192.168.206.119@tcp:12/10 lens 576/1152 e 0 to 1 dl 1779206009 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 4099.090653] Lustre: 50261:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
[ 4108.383538] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 11:53:37 (1779206017)
[ 4135.801493] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 11:54:04 (1779206044)
[ 4161.112912] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 11:54:29 (1779206069)
[ 4161.788927] Lustre: Mounted lustre-client
[ 4193.759211] Lustre: 2365:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779206087/real 1779206087]  req@ffff9dbb85492d80 x1865628564089216/t0(0) o400->MGC192.168.206.119@tcp@192.168.206.119@tcp:26/25 lens 224/224 e 0 to 1 dl 1779206103 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4193.800542] Lustre: 2365:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 4193.806769] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 192.168.206.119@tcp) was lost; in progress operations using this service will fail
[ 4193.821506] LustreError: Skipped 3 previous similar messages
[ 4204.016931] Lustre: Evicted from MGS (at 192.168.206.119@tcp) after server handle changed from 0xda6eebafea75da42 to 0xda6eebafea887d9b
[ 4204.026625] Lustre: Skipped 2 previous similar messages
[ 4224.270215] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4226.695240] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4228.893708] LustreError: 53015:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff9dbb84f27000: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 4228.919571] LustreError: 53015:0:(lov_obd.c:792:lov_cleanup()) Skipped 1 previous similar message
[ 4228.981014] Lustre: Unmounted lustre-client
[ 4235.622231] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 11:55:44 (1779206144)
[ 4358.048185] Lustre: 2367:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779206252/real 1779206253]  req@ffff9dbb85d17480 x1865628564125824/t0(0) o4->lustre-OST0000-osc-ffff9dbb866ee800@192.168.206.119@tcp:6/4 lens 488/448 e 0 to 1 dl 1779206268 ref 2 fl Rpc:XQr/602/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4358.094478] Lustre: 2367:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 62 previous similar messages
[ 4389.855931] Lustre: lustre-OST0000-osc-ffff9dbb866ee800: Connection to lustre-OST0000 (at 192.168.206.119@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4389.884633] Lustre: Skipped 13 previous similar messages
[ 4422.664438] Lustre: lustre-OST0000-osc-ffff9dbb866ee800: Connection restored to 192.168.206.119@tcp (at 192.168.206.119@tcp)
[ 4422.672790] Lustre: Skipped 16 previous similar messages
[ 4536.069897] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 12:00:44 (1779206444)
[ 4587.026505] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 12:01:35 (1779206495)
[ 4591.547296] LustreError: 55186:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff9dbb866ee800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 4591.562990] LustreError: 55186:0:(lov_obd.c:792:lov_cleanup()) Skipped 1 previous similar message
[ 4591.691507] Lustre: Unmounted lustre-client
[ 4617.384953] Lustre: Mounted lustre-client
[ 4625.267189] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 12:02:13 (1779206533)
[ 4625.643748] LustreError: 56167:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4626.707807] LustreError: 56167:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4626.986818] Lustre: *** cfs_fail_loc=305, val=0***
[ 4652.402832] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 12:02:41 (1779206561)
[ 4652.934900] Lustre: Mounted lustre-client
[ 4655.957782] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4666.305909] Lustre: Unmounted lustre-client
[ 4675.438332] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 12:03:03 (1779206583)
[ 4789.122456] LustreError: lustre-MDT0000-mdc-ffff9dbb90485000: operation mds_reint to node 192.168.206.119@tcp failed: rc = -19
[ 4789.136324] LustreError: Skipped 1 previous similar message
[ 4807.647222] Lustre: 2367:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779206701/real 1779206701]  req@ffff9dbb86d90700 x1865628566796928/t0(0) o400->MGC192.168.206.119@tcp@192.168.206.119@tcp:26/25 lens 224/224 e 0 to 1 dl 1779206717 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4807.697770] Lustre: 2367:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 85 previous similar messages
[ 4807.713566] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 192.168.206.119@tcp) was lost; in progress operations using this service will fail
[ 4816.939682] Lustre: Evicted from MGS (at 192.168.206.119@tcp) after server handle changed from 0xda6eebafea889606 to 0xda6eebafea8b4675
[ 5002.081764] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 12:08:30 (1779206910)
[ 5007.743445] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5021.170990] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 192.168.206.119@tcp) was lost; in progress operations using this service will fail
[ 5021.173442] Lustre: lustre-MDT0000-mdc-ffff9dbb90485000: Connection to lustre-MDT0000 (at 192.168.206.119@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5021.209408] Lustre: Skipped 11 previous similar messages
[ 5021.229404] Lustre: Evicted from MGS (at 192.168.206.119@tcp) after server handle changed from 0xda6eebafea8b4675 to 0xda6eebafea901843
[ 5021.265989] LustreError: lustre-MDT0000-mdc-ffff9dbb90485000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5038.060913] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 12:09:06 (1779206946)
[ 5038.655518] Lustre: Mounted lustre-client
[ 5046.349261] LustreError: lustre-OST0000-osc-ffff9dbb90485000: operation ldlm_enqueue to node 192.168.206.119@tcp failed: rc = -107
[ 5046.366379] LustreError: lustre-OST0000-osc-ffff9dbb90485000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5056.516952] Lustre: lustre-OST0000-osc-ffff9dbb84f22800: Connection restored to 192.168.206.119@tcp (at 192.168.206.119@tcp)
[ 5056.535438] Lustre: Skipped 12 previous similar messages
[ 5058.811226] LustreError: 59906:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff9dbb84f22800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 5058.836913] LustreError: 59906:0:(lov_obd.c:792:lov_cleanup()) Skipped 1 previous similar message
[ 5058.924775] Lustre: Unmounted lustre-client
[ 5065.610689] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 12:09:34 (1779206974)
[ 5072.630969] LustreError: lustre-MDT0000-mdc-ffff9dbb90485000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5072.653385] LustreError: 60549:0:(file.c:6098:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 5080.785225] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 12:09:49 (1779206989)
[ 5081.021472] LustreError: 61253:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 5084.996087] LustreError: lustre-MDT0000-mdc-ffff9dbb90485000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5085.011498] LustreError: 61269:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 5085.024330] LustreError: 61253:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=1002
[ 5085.028188] LustreError: 61253:0:(import.c:719:ptlrpc_connect_import_locked()) already connecting
[ 5085.058222] LustreError: 61274:0:(file.c:6098:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5086.194952] LustreError: 61281:0:(file.c:6098:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5086.219241] LustreError: 61281:0:(file.c:6098:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5088.500452] LustreError: 61303:0:(file.c:6098:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5088.511045] LustreError: 61303:0:(file.c:6098:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5093.148965] LustreError: 61350:0:(file.c:6098:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5093.167955] LustreError: 61350:0:(file.c:6098:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 5103.410510] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 12:10:12 (1779207012)
[ 5140.734408] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5142.304464] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5152.621703] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 12:11:01 (1779207061)
[ 5187.670769] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5189.267114] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5200.065618] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 12:11:48 (1779207108)
[ 5266.263464] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5268.278400] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5278.135971] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 12:13:06 (1779207186)
[ 5316.306835] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5318.477361] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5325.824304] LustreError: 66914:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff9dbb90485000: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 5325.840724] LustreError: 66914:0:(lov_obd.c:792:lov_cleanup()) Skipped 1 previous similar message
[ 5325.908123] Lustre: Unmounted lustre-client
[ 5379.595821] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5382.358408] Lustre: Mounted lustre-client
[ 5392.740679] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 12:15:01 (1779207301)
[ 5396.088476] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5398.059410] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 12:15:06 (1779207306)
[ 5434.103699] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 12:15:42 (1779207342)
[ 5436.023136] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5438.072691] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 12:15:46 (1779207346)
[ 5439.554640] Lustre: *** cfs_fail_loc=805, val=0***
[ 5439.621933] Lustre: Mounted lustre-client
[ 5446.385988] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5466.597147] Lustre: 2367:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779207360/real 1779207360]  req@ffff9dbb83104000 x1865628568860416/t0(0) o400->lustre-MDT0000-mdc-ffff9dbb9245c800@192.168.206.119@tcp:12/10 lens 224/224 e 0 to 1 dl 1779207376 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5466.631474] Lustre: 2367:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5466.642127] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 192.168.206.119@tcp) was lost; in progress operations using this service will fail
[ 5475.815387] Lustre: Evicted from MGS (at 192.168.206.119@tcp) after server handle changed from 0xda6eebafea90225a to 0xda6eebafea902604
[ 5478.644170] LustreError: 70251:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff9dbb84f27000: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 5478.664750] LustreError: 70251:0:(lov_obd.c:792:lov_cleanup()) Skipped 1 previous similar message
[ 5478.717294] Lustre: Unmounted lustre-client
[ 5487.033925] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 12:16:35 (1779207395)
[ 5529.114747] LustreError: 2363:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff9dbb86ccd500 x1865628568871680/t90194313220(90194313220) o101->lustre-MDT0000-mdc-ffff9dbb9245c800@192.168.206.119@tcp:12/10 lens 664/608 e 0 to 0 dl 1779207455 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5534.821512] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5536.946876] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5547.873815] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 12:17:36 (1779207456)
[ 5549.242764] LustreError: lustre-OST0000-osc-ffff9dbb9245c800: operation ost_write to node 192.168.206.119@tcp failed: rc = -107
[ 5549.260883] LustreError: Skipped 2 previous similar messages
[ 5549.288283] LustreError: lustre-OST0000-osc-ffff9dbb9245c800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5549.310418] Lustre: 2365:0:(llite_lib.c:4187:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.206.119@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5558.457849] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 12:17:47 (1779207467)
[ 5560.260731] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5562.334509] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 12:17:50 (1779207470)
[ 5563.617555] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5565.719990] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 12:17:54 (1779207474)
[ 5567.605172] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5569.717953] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 12:17:58 (1779207478)
[ 5571.202333] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5573.455820] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 12:18:01 (1779207481)
[ 5575.268524] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5577.655490] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 12:18:05 (1779207485)
[ 5579.528733] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5581.574709] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 12:18:10 (1779207490)
[ 5583.469074] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5585.563067] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 12:18:14 (1779207494)
[ 5587.437910] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5589.663521] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 12:18:18 (1779207498)
[ 5591.670537] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5593.857686] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 12:18:22 (1779207502)
[ 5595.870804] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5597.993712] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 12:18:26 (1779207506)
[ 5599.954488] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5602.027405] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 12:18:30 (1779207510)
[ 5604.840720] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5606.963704] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 12:18:35 (1779207515)
[ 5626.343126] Lustre: lustre-MDT0000-mdc-ffff9dbb9245c800: Connection to lustre-MDT0000 (at 192.168.206.119@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5626.360741] Lustre: Skipped 14 previous similar messages
[ 5639.909806] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 12:19:08 (1779207548)
[ 5674.120730] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 12:19:41 (1779207581)
[ 5675.004640] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5688.505172] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 12:19:56 (1779207596)
[ 5689.015259] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5700.481340] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 12:20:09 (1779207609)
[ 5701.026522] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5710.299863] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 12:20:18 (1779207618)
[ 5710.745039] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5719.208288] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 12:20:28 (1779207628)
[ 5719.492061] Lustre: *** cfs_fail_loc=510, val=3***
[ 5728.821902] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 12:20:37 (1779207637)
[ 5729.359261] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5740.496771] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 12:20:48 (1779207648)
[ 5740.852433] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5809.125444] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 12:21:57 (1779207717)
[ 5809.816456] LustreError: 82438:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5813.030761] LustreError: lustre-MDT0000-mdc-ffff9dbb9245c800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5813.070165] LustreError: 82452:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9dbb9245c800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9dbbc20da000) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5813.101788] Lustre: lustre-MDT0000-mdc-ffff9dbb9245c800: Connection restored to 192.168.206.119@tcp (at 192.168.206.119@tcp)
[ 5813.113098] Lustre: Skipped 14 previous similar messages
[ 5813.895543] LustreError: 82438:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5816.088349] LustreError: 82468:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5819.027712] LustreError: lustre-MDT0000-mdc-ffff9dbb9245c800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5819.052493] LustreError: 82482:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9dbb9245c800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9dbb86c94f00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5819.096492] LustreError: 82482:0:(ldlm_resource.c:1180:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5820.167663] LustreError: 82468:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5827.267860] LustreError: lustre-MDT0000-mdc-ffff9dbb9245c800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5827.825928] LustreError: 82523:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5830.923549] LustreError: lustre-MDT0000-mdc-ffff9dbb9245c800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5831.009606] LustreError: 82543:0:(file.c:6098:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5831.022655] LustreError: 82543:0:(file.c:6098:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5831.911133] LustreError: 82523:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5831.925230] LustreError: 82523:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff9dbb9245c800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5832.136422] LustreError: 82549:0:(file.c:6098:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5832.150407] LustreError: 82549:0:(file.c:6098:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5834.541776] LustreError: 82572:0:(file.c:6098:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5834.554911] LustreError: 82572:0:(file.c:6098:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5838.104725] LustreError: 82598:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5841.489446] LustreError: lustre-MDT0000-mdc-ffff9dbb9245c800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5841.510610] LustreError: 82613:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9dbb9245c800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9dbb879a1800) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5841.542537] LustreError: 82613:0:(ldlm_resource.c:1180:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5842.183132] LustreError: 82598:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5849.524102] LustreError: lustre-MDT0000-mdc-ffff9dbb9245c800: operation ldlm_enqueue to node 192.168.206.119@tcp failed: rc = -107
[ 5849.534600] LustreError: Skipped 5 previous similar messages
[ 5849.556861] LustreError: lustre-MDT0000-mdc-ffff9dbb9245c800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5850.214335] LustreError: 82655:0:(ldlm_flock.c:858:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5853.392429] LustreError: lustre-MDT0000-mdc-ffff9dbb9245c800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5853.425034] LustreError: 82669:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9dbb9245c800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9dbb879a1c00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5853.461529] LustreError: 82669:0:(ldlm_resource.c:1180:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5854.218402] LustreError: 82655:0:(ldlm_flock.c:858:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5857.452128] LustreError: lustre-MDT0000-mdc-ffff9dbb9245c800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5857.576329] LustreError: 82702:0:(file.c:6098:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5857.601548] LustreError: 82702:0:(file.c:6098:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5858.458739] LustreError: 82683:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff9dbb9245c800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5866.739504] LustreError: 82754:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5866.756426] LustreError: 82754:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5869.981455] LustreError: lustre-MDT0000-mdc-ffff9dbb9245c800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5870.039591] LustreError: 82777:0:(file.c:6098:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5870.048933] LustreError: 82777:0:(file.c:6098:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5870.831107] LustreError: 82754:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5870.840822] LustreError: 82754:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5870.864380] LustreError: 82754:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff9dbb9245c800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5882.048575] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 12:23:10 (1779207790)
[ 5900.733540] LustreError: 79189:0:(ldlm_lockd.c:2883:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5900.751649] LustreError: 79189:0:(ldlm_lockd.c:2883:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5904.823339] LustreError: 79189:0:(ldlm_lockd.c:2883:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5904.831846] LustreError: 79189:0:(ldlm_lockd.c:2883:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5911.552793] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 12:23:40 (1779207820)
[ 5964.710611] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 12:24:32 (1779207872)
[ 6028.476882] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 12:25:37 (1779207937)
[ 6063.277876] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 12:26:11 (1779207971)
[ 6063.839758] Lustre: Mounted lustre-client
[ 6188.178586] LustreError: 86462:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff9dbb90486000: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 6188.192302] LustreError: 86462:0:(lov_obd.c:792:lov_cleanup()) Skipped 1 previous similar message
[ 6188.276399] Lustre: Unmounted lustre-client
[ 6196.463727] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 12:28:25 (1779208105)
[ 6197.245210] LustreError: 87077:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 6199.954058] LustreError: lustre-OST0000-osc-ffff9dbb9245c800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6199.974267] LustreError: 87159:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-OST0000-osc-ffff9dbb9245c800: namespace resource [0x240000400:0x39cd:0x0].0x0 (ffff9dbb87ce1600) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6200.005307] LustreError: 87159:0:(ldlm_resource.c:1180:ldlm_resource_complain()) Skipped 1 previous similar message
[ 6200.383176] LustreError: 87077:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout interrupted
[ 6200.400441] Lustre: 2364:0:(llite_lib.c:4187:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.206.119@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 6207.169298] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 12:28:36 (1779208116)
[ 6264.799992] Lustre: 87744:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779208120/real 1779208120]  req@ffff9dbb905c6300 x1865628569046528/t0(0) o101->lustre-MDT0000-mdc-ffff9dbb9245c800@192.168.206.119@tcp:12/10 lens 328/344 e 0 to 1 dl 1779208175 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 6264.839952] Lustre: 87744:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 18 previous similar messages
[ 6264.848627] Lustre: lustre-MDT0000-mdc-ffff9dbb9245c800: Connection to lustre-MDT0000 (at 192.168.206.119@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6264.864214] Lustre: Skipped 14 previous similar messages
[ 6272.338284] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 12:29:41 (1779208181)
[ 6273.918430] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 6275.822612] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 12:29:44 (1779208184)
[ 6339.511301] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6341.314195] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 12:30:50 (1779208250)
[ 6399.877320] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 12:31:48 (1779208308)
[ 6403.031567] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6405.127439] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 12:31:53 (1779208313)
[ 6406.795955] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6408.774484] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 12:31:57 (1779208317)
[ 6446.893833] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 12:32:35 (1779208355)
[ 6459.382929] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6484.959279] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 192.168.206.119@tcp) was lost; in progress operations using this service will fail
[ 6484.980495] LustreError: Skipped 2 previous similar messages
[ 6490.255691] Lustre: lustre-MDT0000-mdc-ffff9dbb9245c800: Connection restored to 192.168.206.119@tcp (at 192.168.206.119@tcp)
[ 6490.269338] Lustre: Skipped 14 previous similar messages
[ 6495.274078] Lustre: Evicted from MGS (at 192.168.206.119@tcp) after server handle changed from 0xda6eebafea9030a0 to 0xda6eebafea904b6c
[ 6495.284410] Lustre: Skipped 2 previous similar messages
[ 6500.326923] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6502.047992] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6512.579284] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 12:33:40 (1779208420)
[ 6515.353666] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6517.841337] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 12:33:46 (1779208426)
[ 6531.051513] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 192.168.206.119@tcp) was lost; in progress operations using this service will fail
[ 6531.068897] Lustre: Evicted from MGS (at 192.168.206.119@tcp) after server handle changed from 0xda6eebafea904b6c to 0xda6eebafea904d87
[ 6546.173360] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 12:34:14 (1779208454)
[ 6598.038066] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 12:35:06 (1779208506)
[ 6604.969235] LustreError: lustre-OST0000-osc-ffff9dbb9245c800: operation ost_setattr to node 192.168.206.119@tcp failed: rc = -19
[ 6604.974628] LustreError: Skipped 4 previous similar messages
[ 6655.166084] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 6660.841347] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6746.399243] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 192.168.206.119@tcp) was lost; in progress operations using this service will fail
[ 6746.414880] LustreError: Skipped 1 previous similar message
[ 6755.889430] Lustre: Evicted from MGS (at 192.168.206.119@tcp) after server handle changed from 0xda6eebafea904f94 to 0xda6eebafea9060c7
[ 6755.906934] Lustre: Skipped 1 previous similar message
[ 6761.312675] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6762.955367] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6797.814844] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6799.434215] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6935.863184] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 12:40:44 (1779208844)
[ 6947.952310] Lustre: lustre-OST0000-osc-ffff9dbb9245c800: Connection to lustre-OST0000 (at 192.168.206.119@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6947.969199] Lustre: Skipped 8 previous similar messages
[ 6993.791459] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 6997.248627] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 7252.460923] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 12:46:01 (1779209161)
[ 7406.056110] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 192.168.206.119@tcp) was lost; in progress operations using this service will fail
[ 7406.072545] LustreError: Skipped 1 previous similar message
[ 7406.104961] Lustre: Evicted from MGS (at 192.168.206.119@tcp) after server handle changed from 0xda6eebafea906209 to 0xda6eebafeaa0049d
[ 7406.117532] Lustre: Skipped 1 previous similar message
[ 7406.124852] Lustre: MGC192.168.206.119@tcp: Connection restored to 192.168.206.119@tcp (at 192.168.206.119@tcp)
[ 7406.132992] Lustre: Skipped 9 previous similar messages
[ 7432.671169] Lustre: 2365:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779209306/real 1779209306]  req@ffff9dbb880f7100 x1865628586771968/t0(0) o400->lustre-MDT0000-mdc-ffff9dbb9245c800@192.168.206.119@tcp:12/10 lens 224/224 e 0 to 1 dl 1779209342 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7432.700238] Lustre: 2365:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 23 previous similar messages
[ 7440.661759] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 12:49:09 (1779209349)
[ 7442.084547] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7443.550302] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 12:49:12 (1779209352)
[ 7445.534945] LustreError: lustre-MDT0000-mdc-ffff9dbb9245c800: operation ldlm_enqueue to node 192.168.206.119@tcp failed: rc = -107
[ 7445.550083] LustreError: Skipped 368 previous similar messages
[ 7445.573488] LustreError: lustre-MDT0000-mdc-ffff9dbb9245c800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7452.813595] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 12:49:21 (1779209361)
[ 7620.532645] LustreError: lustre-OST0000-osc-ffff9dbb9245c800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7622.298689] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 12:52:11 (1779209531)
[ 7648.226033] Lustre: lustre-OST0000-osc-ffff9dbb9245c800: Connection to lustre-OST0000 (at 192.168.206.119@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7648.255734] Lustre: Skipped 3 previous similar messages
[ 7664.675800] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 12:52:53 (1779209573)
[ 7666.088341] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7667.825530] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 12:52:56 (1779209576)
[ 7669.176765] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7670.885279] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 12:52:59 (1779209579)
[ 7703.167148] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 12:53:31 (1779209611)
[ 7740.410434] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 192.168.206.119@tcp) was lost; in progress operations using this service will fail
[ 7740.439589] Lustre: Evicted from MGS (at 192.168.206.119@tcp) after server handle changed from 0xda6eebafeaa0049d to 0xda6eebafeaa02e49
[ 7740.448771] Lustre: Skipped 1 previous similar message
[ 7754.923829] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 12:54:23 (1779209663)
[ 7756.382660] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7758.373485] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 12:54:26 (1779209666)
[ 7759.853664] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7761.639372] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 12:54:30 (1779209670)
[ 7776.408438] LustreError: 105978:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff9dbb9245c800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 7776.415239] LustreError: 105978:0:(lov_obd.c:792:lov_cleanup()) Skipped 1 previous similar message
[ 7776.533865] Lustre: Unmounted lustre-client
[ 7778.236296] Lustre: Mounted lustre-client
[ 7780.729332] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7811.174349] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2