[ 1965.630520] LustreError: lustre-OST0000-osc-ffff96801219d000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1965.650308] Lustre: 2354:0:(llite_lib.c:4366:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.121@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1971.176698] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 09:24:38 (1789133078)
[ 1972.918434] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1974.598984] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 09:24:41 (1789133081)
[ 1976.162456] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1978.274160] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 09:24:44 (1789133084)
[ 1981.495445] LustreError: lustre-MDT0000-mdc-ffff96801219d000: operation mds_reint to node 192.168.204.121@tcp failed: rc = -19
[ 1981.500833] LustreError: Skipped 2 previous similar messages
[ 2000.159230] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 192.168.204.121@tcp) was lost; in progress operations using this service will fail
[ 2010.344893] Lustre: Evicted from MGS (at 192.168.204.121@tcp) after server handle changed from 0xe82edf2f50e15403 to 0xe82edf2f50e16eac
[ 2111.417110] LustreError: lustre-MDT0000-mdc-ffff96801219d000: operation mds_reint to node 192.168.204.121@tcp failed: rc = -19
[ 2128.351662] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 192.168.204.121@tcp) was lost; in progress operations using this service will fail
[ 2138.674619] Lustre: Evicted from MGS (at 192.168.204.121@tcp) after server handle changed from 0xe82edf2f50e16eac to 0xe82edf2f50e373c5
[ 2145.290708] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 09:27:31 (1789133251)
[ 2145.756461] Lustre: *** cfs_fail_loc=305, val=0***
[ 2145.757922] Lustre: Skipped 2 previous similar messages
[ 2190.815764] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 192.168.204.121@tcp) was lost; in progress operations using this service will fail
[ 2190.872817] Lustre: Evicted from MGS (at 192.168.204.121@tcp) after server handle changed from 0xe82edf2f50e373c5 to 0xe82edf2f50e379b4
[ 2204.342365] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2205.926458] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2215.000907] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 09:28:41 (1789133321)
[ 2230.786139] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 192.168.204.121@tcp) was lost; in progress operations using this service will fail
[ 2230.813904] Lustre: lustre-MDT0000-mdc-ffff96801219d000: Connection to lustre-MDT0000 (at 192.168.204.121@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2230.831636] Lustre: Evicted from MGS (at 192.168.204.121@tcp) after server handle changed from 0xe82edf2f50e379b4 to 0xe82edf2f50e37e6f
[ 2230.848859] Lustre: Skipped 12 previous similar messages
[ 2230.876418] Lustre: MGC192.168.204.121@tcp: Connection restored to 192.168.204.121@tcp (at 192.168.204.121@tcp)
[ 2230.893986] Lustre: Skipped 15 previous similar messages
[ 2230.949913] LustreError: lustre-MDT0000-mdc-ffff96801219d000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2254.112983] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 09:29:20 (1789133360)
[ 2286.478727] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 09:29:52 (1789133392)
[ 2299.552391] LustreError: lustre-MDT0000-mdc-ffff96801219d000: operation mds_reint to node 192.168.204.121@tcp failed: rc = -19
[ 2299.562800] LustreError: Skipped 2 previous similar messages
[ 2318.881476] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 192.168.204.121@tcp) was lost; in progress operations using this service will fail
[ 2328.255785] Lustre: Evicted from MGS (at 192.168.204.121@tcp) after server handle changed from 0xe82edf2f50e37e6f to 0xe82edf2f50e3d46e
[ 2350.583284] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2352.964441] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2439.664615] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 192.168.204.121@tcp) was lost; in progress operations using this service will fail
[ 2439.696247] Lustre: Evicted from MGS (at 192.168.204.121@tcp) after server handle changed from 0xe82edf2f50e3d46e to 0xe82edf2f50e59524
[ 2440.959887] Lustre: 16172:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.121@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2467.614093] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2470.342218] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2538.078684] LustreError: lustre-MDT0000-mdc-ffff96801219d000: operation ldlm_enqueue to node 192.168.204.121@tcp failed: rc = -19
[ 2538.089372] LustreError: Skipped 2 previous similar messages
[ 2557.857477] Lustre: 2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789133649/real 1789133649]  req@ffff9680087e7c00 x1876040053912320/t0(0) o400->MGC192.168.204.121@tcp@192.168.204.121@tcp:26/25 lens 224/224 e 0 to 1 dl 1789133665 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2557.875865] Lustre: 2357:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 12 previous similar messages
[ 2557.881680] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 192.168.204.121@tcp) was lost; in progress operations using this service will fail
[ 2567.082167] Lustre: Evicted from MGS (at 192.168.204.121@tcp) after server handle changed from 0xe82edf2f50e59524 to 0xe82edf2f50e705df
[ 2584.454450] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2586.398060] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2615.857428] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 09:35:22 (1789133722)
[ 2638.879910] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 192.168.204.121@tcp) was lost; in progress operations using this service will fail
[ 2648.057206] Lustre: Evicted from MGS (at 192.168.204.121@tcp) after server handle changed from 0xe82edf2f50e705df to 0xe82edf2f50e7d55b
[ 2650.098380] Lustre: 16172:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.121@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2663.619750] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2696.929671] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2742.881665] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2775.071298] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 192.168.204.121@tcp) was lost; in progress operations using this service will fail
[ 2775.103055] LustreError: Skipped 2 previous similar messages
[ 2785.321052] Lustre: Evicted from MGS (at 192.168.204.121@tcp) after server handle changed from 0xe82edf2f50e84acc to 0xe82edf2f50e88d45
[ 2785.339138] Lustre: Skipped 2 previous similar messages
[ 2793.811801] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2817.489564] LustreError: lustre-MDT0000-mdc-ffff96801219d000: operation ldlm_enqueue to node 192.168.204.121@tcp failed: rc = -19
[ 2817.495904] LustreError: Skipped 4 previous similar messages
[ 2847.265066] Lustre: MGC192.168.204.121@tcp: Connection restored to 192.168.204.121@tcp (at 192.168.204.121@tcp)
[ 2847.278500] Lustre: Skipped 15 previous similar messages
[ 2861.062580] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2889.816330] Lustre: lustre-MDT0000-mdc-ffff96801219d000: Connection to lustre-MDT0000 (at 192.168.204.121@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2889.840719] Lustre: Skipped 9 previous similar messages
[ 2919.297134] Lustre: 16172:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.121@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2936.066286] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 3035.067842] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 09:42:21 (1789134141)
[ 3089.559383] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3092.125953] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3383.454539] LustreError: lustre-OST0000-osc-ffff96801219d000: operation ldlm_enqueue to node 192.168.204.121@tcp failed: rc = -107
[ 3383.469199] LustreError: Skipped 4 previous similar messages
[ 3429.141051] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3431.740248] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3715.746422] Lustre: lustre-OST0000-osc-ffff96801219d000: Connection to lustre-OST0000 (at 192.168.204.121@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3715.764265] Lustre: Skipped 3 previous similar messages
[ 3740.411274] Lustre: lustre-OST0000-osc-ffff96801219d000: Connection restored to 192.168.204.121@tcp (at 192.168.204.121@tcp)
[ 3740.423650] Lustre: Skipped 6 previous similar messages
[ 3762.800709] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3766.256161] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4009.739399] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 09:58:36 (1789135116)
[ 4027.871698] Lustre: 50814:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789135119/real 1789135119]  req@ffff968010587480 x1876040072031104/t0(0) o101->lustre-MDT0000-mdc-ffff96801219d000@192.168.204.121@tcp:12/10 lens 576/1152 e 0 to 1 dl 1789135135 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 4027.919095] Lustre: 50814:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
[ 4038.148689] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 09:59:04 (1789135144)
[ 4065.982744] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 09:59:31 (1789135171)
[ 4092.160932] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 09:59:58 (1789135198)
[ 4092.721889] Lustre: Mounted lustre-client - version 2.17.57_132_gd76390e
[ 4124.642918] Lustre: 2354:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789135216/real 1789135216]  req@ffff968010584b40 x1876040072049024/t0(0) o400->lustre-MDT0000-mdc-ffff96801219d000@192.168.204.121@tcp:12/10 lens 224/224 e 0 to 1 dl 1789135232 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4124.690266] Lustre: 2354:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 4124.707299] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 192.168.204.121@tcp) was lost; in progress operations using this service will fail
[ 4124.730286] LustreError: Skipped 3 previous similar messages
[ 4134.908302] Lustre: Evicted from MGS (at 192.168.204.121@tcp) after server handle changed from 0xe82edf2f50ea55bf to 0xe82edf2f50fc4e32
[ 4134.922106] Lustre: Skipped 3 previous similar messages
[ 4136.885922] Lustre: 16172:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.121@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 4156.365491] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4158.081118] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4160.872747] Lustre: Unmounted lustre-client
[ 4169.113994] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 10:01:15 (1789135275)
[ 4275.167233] Lustre: 2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789135367/real 1789135367]  req@ffff96801203c780 x1876040072084096/t0(0) o4->lustre-OST0000-osc-ffff96801219d000@192.168.204.121@tcp:6/4 lens 488/448 e 0 to 1 dl 1789135383 ref 2 fl Rpc:XQr/602/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4275.206851] Lustre: 2355:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 53 previous similar messages
[ 4324.319249] Lustre: lustre-OST0000-osc-ffff96801219d000: Connection to lustre-OST0000 (at 192.168.204.121@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4324.335415] Lustre: Skipped 13 previous similar messages
[ 4340.603607] Lustre: lustre-OST0000-osc-ffff96801219d000: Connection restored to 192.168.204.121@tcp (at 192.168.204.121@tcp)
[ 4340.628661] Lustre: Skipped 15 previous similar messages
[ 4471.092359] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 10:06:16 (1789135576)
[ 4523.346277] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 10:07:09 (1789135629)
[ 4526.834434] Lustre: Unmounted lustre-client
[ 4552.853064] Lustre: Mounted lustre-client - version 2.17.57_132_gd76390e
[ 4563.020154] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 10:07:49 (1789135669)
[ 4563.485814] LustreError: 56758:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4564.554735] LustreError: 56758:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4564.908881] Lustre: *** cfs_fail_loc=305, val=0***
[ 4590.783413] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 10:08:17 (1789135697)
[ 4591.177772] Lustre: Mounted lustre-client - version 2.17.57_132_gd76390e
[ 4594.751182] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4605.066773] Lustre: Unmounted lustre-client
[ 4612.960365] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 10:08:39 (1789135719)
[ 4707.094114] LustreError: lustre-MDT0000-mdc-ffff968007c90800: operation mds_reint to node 192.168.204.121@tcp failed: rc = -19
[ 4707.111755] LustreError: Skipped 1 previous similar message
[ 4725.215743] Lustre: 2356:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789135817/real 1789135817]  req@ffff968005806d00 x1876040074743808/t0(0) o400->MGC192.168.204.121@tcp@192.168.204.121@tcp:26/25 lens 224/224 e 0 to 1 dl 1789135833 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4725.257923] Lustre: 2356:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 84 previous similar messages
[ 4725.267939] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 192.168.204.121@tcp) was lost; in progress operations using this service will fail
[ 4735.480345] Lustre: Evicted from MGS (at 192.168.204.121@tcp) after server handle changed from 0xe82edf2f50fc69c2 to 0xe82edf2f50ff1ac4
[ 4912.844498] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 10:13:39 (1789136019)
[ 4917.649563] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4934.135380] Lustre: lustre-MDT0000-mdc-ffff968007c90800: Connection to lustre-MDT0000 (at 192.168.204.121@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4934.145619] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 192.168.204.121@tcp) was lost; in progress operations using this service will fail
[ 4934.158805] Lustre: Skipped 11 previous similar messages
[ 4934.177725] Lustre: Evicted from MGS (at 192.168.204.121@tcp) after server handle changed from 0xe82edf2f50ff1ac4 to 0xe82edf2f5103ed41
[ 4934.246508] LustreError: lustre-MDT0000-mdc-ffff968007c90800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4950.243369] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 10:14:16 (1789136056)
[ 4950.946521] Lustre: Mounted lustre-client - version 2.17.57_132_gd76390e
[ 4958.204199] LustreError: lustre-OST0000-osc-ffff968007c90800: operation ldlm_enqueue to node 192.168.204.121@tcp failed: rc = -107
[ 4958.232288] LustreError: lustre-OST0000-osc-ffff968007c90800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4958.249158] LustreError: 60489:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff968007c90800: namespace resource [0x240000400:0x39a3:0x0].0x0 (ffff968005a84200) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4968.445064] Lustre: lustre-OST0000-osc-ffff9680120ba800: Connection restored to 192.168.204.121@tcp (at 192.168.204.121@tcp)
[ 4968.450465] Lustre: Skipped 13 previous similar messages
[ 4970.617591] Lustre: Unmounted lustre-client
[ 4977.709223] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 10:14:44 (1789136084)
[ 4984.669463] LustreError: lustre-MDT0000-mdc-ffff968007c90800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4984.685863] LustreError: 61160:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4993.712544] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 10:15:00 (1789136100)
[ 4994.000590] LustreError: 61861:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4997.914149] LustreError: lustre-MDT0000-mdc-ffff968007c90800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4997.947231] LustreError: 61881:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4997.970258] LustreError: 61861:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=1043
[ 4998.003460] LustreError: 61861:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting
[ 4998.016704] LustreError: 61886:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4999.171678] LustreError: 61893:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4999.183968] LustreError: 61893:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5001.736243] LustreError: 61915:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5001.753705] LustreError: 61915:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5006.412928] LustreError: 61960:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5006.422391] LustreError: 61960:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 5017.392074] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 10:15:23 (1789136123)
[ 5057.994273] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5059.934437] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5073.943956] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 10:16:19 (1789136179)
[ 5115.805235] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5117.275496] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5130.162784] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 10:17:16 (1789136236)
[ 5196.230457] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5197.632234] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5206.008825] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 10:18:32 (1789136312)
[ 5245.895825] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5247.667952] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5254.045320] Lustre: Unmounted lustre-client
[ 5311.383808] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5314.388989] Lustre: Mounted lustre-client - version 2.17.57_132_gd76390e
[ 5321.515029] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 10:20:28 (1789136428)
[ 5324.352555] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5325.997926] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 10:20:32 (1789136432)
[ 5358.291876] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 10:21:04 (1789136464)
[ 5359.778098] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5361.325530] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 10:21:08 (1789136468)
[ 5362.827837] Lustre: *** cfs_fail_loc=805, val=0***
[ 5362.893715] Lustre: Mounted lustre-client - version 2.17.57_132_gd76390e
[ 5368.005662] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5389.792824] Lustre: 2356:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789136481/real 1789136481]  req@ffff968008601a40 x1876040076813824/t0(0) o400->MGC192.168.204.121@tcp@192.168.204.121@tcp:26/25 lens 224/224 e 0 to 1 dl 1789136497 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5389.822506] Lustre: 2356:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5389.828174] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 192.168.204.121@tcp) was lost; in progress operations using this service will fail
[ 5389.858157] Lustre: Evicted from MGS (at 192.168.204.121@tcp) after server handle changed from 0xe82edf2f5103f73c to 0xe82edf2f5103fa53
[ 5399.564983] Lustre: Unmounted lustre-client
[ 5407.838599] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 10:21:54 (1789136514)
[ 5435.933587] LustreError: 2353:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff968018dacb40 x1876040076825600/t94489280516(94489280516) o101->lustre-MDT0000-mdc-ffff9680066bd800@192.168.204.121@tcp:12/10 lens 664/608 e 0 to 0 dl 1789136559 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5447.509806] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5449.385963] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5457.885575] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 10:22:44 (1789136564)
[ 5459.337429] LustreError: lustre-OST0000-osc-ffff9680066bd800: operation ost_write to node 192.168.204.121@tcp failed: rc = -107
[ 5459.343986] LustreError: Skipped 2 previous similar messages
[ 5459.359907] LustreError: lustre-OST0000-osc-ffff9680066bd800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5459.373078] Lustre: 2357:0:(llite_lib.c:4366:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.121@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5459.413209] LustreError: 73067:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff9680066bd800: namespace resource [0x240000400:0x3a02:0x0].0x0 (ffff968005b6b900) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5468.914595] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 10:22:55 (1789136575)
[ 5470.442181] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5472.724925] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 10:22:58 (1789136578)
[ 5474.208988] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5476.140451] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 10:23:02 (1789136582)
[ 5477.680733] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5479.214487] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 10:23:05 (1789136585)
[ 5480.805404] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5482.615883] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 10:23:09 (1789136589)
[ 5484.042555] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5485.724343] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 10:23:12 (1789136592)
[ 5487.378421] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5489.114220] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 10:23:15 (1789136595)
[ 5490.694801] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5492.168597] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 10:23:19 (1789136599)
[ 5493.727799] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5495.957544] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 10:23:22 (1789136602)
[ 5497.408760] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5498.990534] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 10:23:25 (1789136605)
[ 5500.583913] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5502.206397] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 10:23:29 (1789136609)
[ 5503.553587] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5505.046195] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 10:23:31 (1789136611)
[ 5507.232387] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5509.189700] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 10:23:35 (1789136615)
[ 5545.083565] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 10:24:11 (1789136651)
[ 5574.297439] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 10:24:41 (1789136681)
[ 5574.758772] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5584.569421] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 10:24:51 (1789136691)
[ 5584.817773] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5594.243760] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 10:25:01 (1789136701)
[ 5594.582738] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5604.181832] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 10:25:10 (1789136710)
[ 5604.450483] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5612.809766] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 10:25:19 (1789136719)
[ 5613.107796] Lustre: *** cfs_fail_loc=510, val=3***
[ 5621.216515] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 10:25:28 (1789136728)
[ 5621.589473] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5630.937823] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 10:25:37 (1789136737)
[ 5631.626684] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5690.783355] Lustre: lustre-OST0000-osc-ffff9680066bd800: Connection to lustre-OST0000 (at 192.168.204.121@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5690.812319] Lustre: Skipped 17 previous similar messages
[ 5699.994145] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 10:26:45 (1789136805)
[ 5700.377315] LustreError: 83277:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5703.569559] LustreError: lustre-MDT0000-mdc-ffff9680066bd800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5703.601419] LustreError: 83291:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9680066bd800: inode [0x20000a042:0x8:0x0] mdc close failed: rc = -108
[ 5703.637037] LustreError: 83291:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9680066bd800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff968005a84700) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5703.658114] LustreError: 83296:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5703.677424] LustreError: 83296:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5703.684549] Lustre: lustre-MDT0000-mdc-ffff9680066bd800: Connection restored to 192.168.204.121@tcp (at 192.168.204.121@tcp)
[ 5703.699607] Lustre: Skipped 16 previous similar messages
[ 5704.463134] LustreError: 83277:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5706.611359] LustreError: 83307:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5709.744138] LustreError: lustre-MDT0000-mdc-ffff9680066bd800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5709.761832] LustreError: 83321:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9680066bd800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff968005b6b000) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5709.791272] LustreError: 83321:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5710.695194] LustreError: 83307:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5717.553049] LustreError: lustre-MDT0000-mdc-ffff9680066bd800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5717.975787] LustreError: 83364:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5720.738484] LustreError: lustre-MDT0000-mdc-ffff9680066bd800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5720.815926] LustreError: 83384:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5722.047200] LustreError: 83364:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5722.063490] LustreError: 83364:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9680066bd800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5722.084053] LustreError: 83364:0:(file.c:251:ll_close_inode_openhandle()) Skipped 1 previous similar message
[ 5723.096808] LustreError: 83401:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5723.114654] LustreError: 83401:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 12 previous similar messages
[ 5727.952275] LustreError: 83439:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5730.971250] LustreError: lustre-MDT0000-mdc-ffff9680066bd800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5731.004128] LustreError: 83453:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9680066bd800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff968005b6b300) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5731.035658] LustreError: 83453:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5732.037555] LustreError: 83439:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5739.194636] LustreError: lustre-MDT0000-mdc-ffff9680066bd800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5739.647083] LustreError: 83495:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5742.406541] LustreError: lustre-MDT0000-mdc-ffff9680066bd800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5742.422373] LustreError: 83509:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9680066bd800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff968021bab100) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5742.439786] LustreError: 83509:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5743.735139] LustreError: 83495:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5746.851819] LustreError: lustre-MDT0000-mdc-ffff9680066bd800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5746.951305] LustreError: 83542:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5746.968878] LustreError: 83542:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5747.953879] LustreError: 83522:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9680066bd800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5755.941915] LustreError: 83593:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5755.952848] LustreError: 83593:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5759.728654] LustreError: lustre-MDT0000-mdc-ffff9680066bd800: operation ldlm_enqueue to node 192.168.204.121@tcp failed: rc = -107
[ 5759.742014] LustreError: Skipped 8 previous similar messages
[ 5759.767285] LustreError: lustre-MDT0000-mdc-ffff9680066bd800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5759.859780] LustreError: 83615:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5759.877569] LustreError: 83615:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5760.031519] LustreError: 83593:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5760.043713] LustreError: 83593:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5760.063612] LustreError: 83593:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9680066bd800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5772.280847] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 10:27:58 (1789136878)
[ 5790.886847] LustreError: 80026:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5790.894914] LustreError: 80026:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5794.959200] LustreError: 80026:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5794.969956] LustreError: 80026:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5801.355273] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 10:28:28 (1789136908)
[ 5852.703143] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 10:29:19 (1789136959)
[ 5916.620402] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 10:30:23 (1789137023)
[ 5950.047979] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 10:30:56 (1789137056)
[ 5950.612936] Lustre: Mounted lustre-client - version 2.17.57_132_gd76390e
[ 6074.679130] Lustre: Unmounted lustre-client
[ 6082.075694] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 10:33:08 (1789137188)
[ 6082.851719] LustreError: 87916:0:(osc_request.c:3192:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 6085.716361] LustreError: lustre-OST0000-osc-ffff9680066bd800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6085.733910] LustreError: 88003:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff9680066bd800: namespace resource [0x240000400:0x3a2d:0x0].0x0 (ffff968005a84200) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6085.759861] LustreError: 88003:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 6085.991164] LustreError: 87916:0:(osc_request.c:3192:osc_build_rpc()) cfs_fail_timeout interrupted
[ 6085.998336] Lustre: 2355:0:(llite_lib.c:4366:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.121@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 6093.671297] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 10:33:19 (1789137199)
[ 6151.135170] Lustre: 88583:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789137204/real 1789137204]  req@ffff968018dad680 x1876040076993920/t0(0) o101->lustre-MDT0000-mdc-ffff9680066bd800@192.168.204.121@tcp:12/10 lens 328/344 e 0 to 1 dl 1789137259 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 6151.193254] Lustre: 88583:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 20 previous similar messages
[ 6159.025133] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 10:34:25 (1789137265)
[ 6160.654565] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 6162.526581] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 10:34:29 (1789137269)
[ 6227.788933] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6229.520729] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 10:35:36 (1789137336)
[ 6288.359370] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 10:36:34 (1789137394)
[ 6290.791614] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6292.802883] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 10:36:39 (1789137399)
[ 6294.777274] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6297.052139] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 10:36:43 (1789137403)
[ 6337.728744] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 10:37:24 (1789137444)
[ 6348.195490] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6373.371347] Lustre: lustre-MDT0000-mdc-ffff9680066bd800: Connection to lustre-MDT0000 (at 192.168.204.121@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6373.375884] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 192.168.204.121@tcp) was lost; in progress operations using this service will fail
[ 6373.395266] Lustre: Skipped 16 previous similar messages
[ 6373.413433] LustreError: Skipped 2 previous similar messages
[ 6373.450283] Lustre: Evicted from MGS (at 192.168.204.121@tcp) after server handle changed from 0xe82edf2f510405ba to 0xe82edf2f51041eb1
[ 6373.463025] Lustre: Skipped 2 previous similar messages
[ 6373.485427] Lustre: MGC192.168.204.121@tcp: Connection restored to 192.168.204.121@tcp (at 192.168.204.121@tcp)
[ 6373.501161] Lustre: Skipped 15 previous similar messages
[ 6391.322503] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6393.652639] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6407.140752] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 10:38:33 (1789137513)
[ 6410.232785] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6412.233031] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 10:38:38 (1789137518)
[ 6424.551256] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 192.168.204.121@tcp) was lost; in progress operations using this service will fail
[ 6424.597341] Lustre: Evicted from MGS (at 192.168.204.121@tcp) after server handle changed from 0xe82edf2f51041eb1 to 0xe82edf2f510421eb
[ 6439.899614] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 10:39:06 (1789137546)
[ 6491.754057] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 10:39:58 (1789137598)
[ 6499.037405] LustreError: lustre-OST0000-osc-ffff9680066bd800: operation ost_setattr to node 192.168.204.121@tcp failed: rc = -107
[ 6499.064604] LustreError: Skipped 4 previous similar messages
[ 6554.515620] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6558.305110] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6645.727352] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 192.168.204.121@tcp) was lost; in progress operations using this service will fail
[ 6645.735726] LustreError: Skipped 1 previous similar message
[ 6655.081651] Lustre: Evicted from MGS (at 192.168.204.121@tcp) after server handle changed from 0xe82edf2f51042572 to 0xe82edf2f51043689
[ 6655.094866] Lustre: Skipped 1 previous similar message
[ 6662.103268] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6663.603539] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6701.725089] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6703.714029] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6839.296301] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 10:45:45 (1789137945)
[ 6908.823337] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6912.053247] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 7189.705463] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 10:51:36 (1789138296)
[ 7326.192568] Lustre: lustre-MDT0000-mdc-ffff9680066bd800: Connection to lustre-MDT0000 (at 192.168.204.121@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7326.213548] Lustre: Skipped 6 previous similar messages
[ 7341.552256] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 192.168.204.121@tcp) was lost; in progress operations using this service will fail
[ 7341.570240] LustreError: Skipped 1 previous similar message
[ 7341.596702] Lustre: Evicted from MGS (at 192.168.204.121@tcp) after server handle changed from 0xe82edf2f5104385e to 0xe82edf2f5113dac1
[ 7341.624544] Lustre: Skipped 1 previous similar message
[ 7341.635420] Lustre: MGC192.168.204.121@tcp: Connection restored to 192.168.204.121@tcp (at 192.168.204.121@tcp)
[ 7341.650507] Lustre: Skipped 9 previous similar messages
[ 7378.447077] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 10:54:45 (1789138485)
[ 7380.223354] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7382.257893] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 10:54:48 (1789138488)
[ 7384.576580] LustreError: lustre-MDT0000-mdc-ffff9680066bd800: operation ldlm_enqueue to node 192.168.204.121@tcp failed: rc = -107
[ 7384.595086] LustreError: Skipped 371 previous similar messages
[ 7384.603917] LustreError: lustre-MDT0000-mdc-ffff9680066bd800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7392.446874] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 10:54:59 (1789138499)
[ 7560.139553] LustreError: lustre-OST0000-osc-ffff9680066bd800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7562.145214] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 10:57:48 (1789138668)
[ 7587.807628] Lustre: 2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789138675/real 1789138675]  req@ffff968010de4b40 x1876040094742144/t0(0) o4->lustre-OST0000-osc-ffff9680066bd800@192.168.204.121@tcp:6/4 lens 4584/448 e 0 to 1 dl 1789138695 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7587.846842] Lustre: 2355:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 23 previous similar messages
[ 7604.626642] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 10:58:31 (1789138711)
[ 7605.956549] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7607.519522] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 10:58:34 (1789138714)
[ 7608.951418] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7610.624802] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 10:58:37 (1789138717)
[ 7644.165945] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 10:59:11 (1789138751)
[ 7683.054949] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 192.168.204.121@tcp) was lost; in progress operations using this service will fail
[ 7683.079982] Lustre: Evicted from MGS (at 192.168.204.121@tcp) after server handle changed from 0xe82edf2f5113dac1 to 0xe82edf2f5113fd89
[ 7683.098133] Lustre: Skipped 1 previous similar message
[ 7701.253872] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 11:00:07 (1789138807)
[ 7702.966351] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7705.616842] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 11:00:11 (1789138811)
[ 7707.599987] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7709.663130] Lustre: 2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789138780/real 1789138780]  req@ffff968026680b40 x1876040094881920/t0(0) o400->lustre-MDT0000-mdc-ffff9680066bd800@192.168.204.121@tcp:12/10 lens 224/224 e 0 to 1 dl 1789138817 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7709.711608] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 11:00:16 (1789138816)
[ 7720.291738] Lustre: Unmounted lustre-client
[ 7722.741973] Lustre: Mounted lustre-client - version 2.17.57_132_gd76390e
[ 7725.647371] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7764.512637] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2