[ 1895.783928] LustreError: lustre-OST0000-osc-ffff93d3510b8000: operation ost_sync to node 192.168.203.139@tcp failed: rc = -107
[ 1895.810569] LustreError: lustre-OST0000-osc-ffff93d3510b8000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1895.828501] Lustre: 2350:0:(llite_lib.c:4366:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.139@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1902.430780] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 10:40:48 (1789137648)
[ 1904.475739] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1906.112161] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 10:40:51 (1789137651)
[ 1907.658549] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1909.437662] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 10:40:55 (1789137655)
[ 1913.403755] LustreError: lustre-MDT0000-mdc-ffff93d3510b8000: operation mds_reint to node 192.168.203.139@tcp failed: rc = -19
[ 1933.298805] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 192.168.203.139@tcp) was lost; in progress operations using this service will fail
[ 1933.334284] Lustre: Evicted from MGS (at 192.168.203.139@tcp) after server handle changed from 0x42304a04bce4f503 to 0x42304a04bce50dd7
[ 1934.385819] Lustre: 16125:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.139@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2042.340181] LustreError: lustre-MDT0000-mdc-ffff93d3510b8000: operation ldlm_enqueue to node 192.168.203.139@tcp failed: rc = -19
[ 2062.303394] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 192.168.203.139@tcp) was lost; in progress operations using this service will fail
[ 2062.328529] Lustre: Evicted from MGS (at 192.168.203.139@tcp) after server handle changed from 0x42304a04bce50dd7 to 0x42304a04bce7041e
[ 2090.190759] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 10:43:55 (1789137835)
[ 2090.806766] Lustre: *** cfs_fail_loc=305, val=0***
[ 2090.808786] Lustre: Skipped 2 previous similar messages
[ 2111.800945] LustreError: lustre-OST0000-osc-ffff93d3510b8000: operation ost_connect to node 192.168.203.139@tcp failed: rc = -75
[ 2132.961844] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 192.168.203.139@tcp) was lost; in progress operations using this service will fail
[ 2132.983930] Lustre: Evicted from MGS (at 192.168.203.139@tcp) after server handle changed from 0x42304a04bce7041e to 0x42304a04bce70a22
[ 2145.669188] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2147.010936] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2155.051771] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 10:45:00 (1789137900)
[ 2168.814740] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 192.168.203.139@tcp) was lost; in progress operations using this service will fail
[ 2168.826845] Lustre: lustre-MDT0000-mdc-ffff93d3510b8000: Connection to lustre-MDT0000 (at 192.168.203.139@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2168.833716] Lustre: Evicted from MGS (at 192.168.203.139@tcp) after server handle changed from 0x42304a04bce70a22 to 0x42304a04bce70f15
[ 2168.851948] Lustre: Skipped 12 previous similar messages
[ 2168.935363] LustreError: lustre-MDT0000-mdc-ffff93d3510b8000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2189.523816] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 10:45:35 (1789137935)
[ 2220.343732] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 10:46:06 (1789137966)
[ 2251.615291] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 192.168.203.139@tcp) was lost; in progress operations using this service will fail
[ 2260.849319] Lustre: Evicted from MGS (at 192.168.203.139@tcp) after server handle changed from 0x42304a04bce70f15 to 0x42304a04bce754dd
[ 2260.880019] Lustre: MGC192.168.203.139@tcp: Connection restored to 192.168.203.139@tcp (at 192.168.203.139@tcp)
[ 2260.903849] Lustre: Skipped 16 previous similar messages
[ 2278.713836] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2280.539177] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2345.550718] LustreError: lustre-MDT0000-mdc-ffff93d3510b8000: operation mds_reint to node 192.168.203.139@tcp failed: rc = -19
[ 2345.569428] LustreError: Skipped 3 previous similar messages
[ 2363.487185] Lustre: 2349:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789138094/real 1789138094]  req@ffff93d347b4f480 x1876044880215040/t0(0) o400->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 224/224 e 0 to 1 dl 1789138110 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2363.522995] Lustre: 2349:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
[ 2363.540208] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 192.168.203.139@tcp) was lost; in progress operations using this service will fail
[ 2373.740170] Lustre: Evicted from MGS (at 192.168.203.139@tcp) after server handle changed from 0x42304a04bce754dd to 0x42304a04bce91569
[ 2389.799417] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2391.980032] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2477.551574] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 192.168.203.139@tcp) was lost; in progress operations using this service will fail
[ 2477.587258] Lustre: Evicted from MGS (at 192.168.203.139@tcp) after server handle changed from 0x42304a04bce91569 to 0x42304a04bcea8911
[ 2479.093536] Lustre: 16125:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.139@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2504.681378] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2506.949575] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2536.960972] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 10:51:22 (1789138282)
[ 2563.562190] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 192.168.203.139@tcp) was lost; in progress operations using this service will fail
[ 2563.577422] Lustre: Evicted from MGS (at 192.168.203.139@tcp) after server handle changed from 0x42304a04bcea8911 to 0x42304a04bceb47ca
[ 2571.274538] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2597.319744] Lustre: 16125:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.139@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2613.460847] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2621.518760] LustreError: lustre-MDT0000-mdc-ffff93d3510b8000: operation ldlm_enqueue to node 192.168.203.139@tcp failed: rc = -19
[ 2621.530889] LustreError: Skipped 4 previous similar messages
[ 2649.784520] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2692.009964] Lustre: Evicted from MGS (at 192.168.203.139@tcp) after server handle changed from 0x42304a04bceb92d2 to 0x42304a04bcec0d8a
[ 2692.029608] Lustre: Skipped 2 previous similar messages
[ 2699.892540] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2743.263401] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 192.168.203.139@tcp) was lost; in progress operations using this service will fail
[ 2743.300343] LustreError: Skipped 3 previous similar messages
[ 2762.426976] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2790.986214] Lustre: lustre-MDT0000-mdc-ffff93d3510b8000: Connection to lustre-MDT0000 (at 192.168.203.139@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2791.008130] Lustre: Skipped 9 previous similar messages
[ 2829.201201] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2881.458952] Lustre: MGC192.168.203.139@tcp: Connection restored to 192.168.203.139@tcp (at 192.168.203.139@tcp)
[ 2881.475839] Lustre: Skipped 17 previous similar messages
[ 2916.225218] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 10:57:42 (1789138662)
[ 2978.836448] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 2981.784801] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3261.583168] LustreError: lustre-OST0000-osc-ffff93d3510b8000: operation ldlm_enqueue to node 192.168.203.139@tcp failed: rc = -19
[ 3261.603299] LustreError: Skipped 8 previous similar messages
[ 3302.930912] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3304.864965] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3593.811195] Lustre: lustre-OST0000-osc-ffff93d3510b8000: Connection to lustre-OST0000 (at 192.168.203.139@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3593.844509] Lustre: Skipped 3 previous similar messages
[ 3619.283574] Lustre: lustre-OST0000-osc-ffff93d3510b8000: Connection restored to 192.168.203.139@tcp (at 192.168.203.139@tcp)
[ 3619.296779] Lustre: Skipped 2 previous similar messages
[ 3638.953896] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3641.277138] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3888.314896] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 11:13:53 (1789139633)
[ 3904.991172] Lustre: 50777:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789139636/real 1789139636]  req@ffff93d3502bb0c0 x1876044900018432/t0(0) o101->lustre-MDT0000-mdc-ffff93d3510b8000@192.168.203.139@tcp:12/10 lens 576/1152 e 0 to 1 dl 1789139652 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3905.030755] Lustre: 50777:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
[ 3912.224515] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 11:14:18 (1789139658)
[ 3938.076849] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 11:14:43 (1789139683)
[ 3963.350995] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 11:15:08 (1789139708)
[ 3963.897234] Lustre: Mounted lustre-client - version 2.17.57_127_ged01a19
[ 3995.615380] Lustre: 2350:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789139726/real 1789139726]  req@ffff93d347afb480 x1876044900038144/t0(0) o400->lustre-MDT0000-mdc-ffff93d34301a800@192.168.203.139@tcp:12/10 lens 224/224 e 0 to 1 dl 1789139742 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3995.647751] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 192.168.203.139@tcp) was lost; in progress operations using this service will fail
[ 3995.661329] Lustre: 2350:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 3995.689040] LustreError: Skipped 2 previous similar messages
[ 3995.703734] Lustre: Evicted from MGS (at 192.168.203.139@tcp) after server handle changed from 0x42304a04bcedd038 to 0x42304a04bd005083
[ 3995.716251] Lustre: Skipped 3 previous similar messages
[ 4009.401759] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4010.806446] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4012.927286] Lustre: Unmounted lustre-client
[ 4018.747311] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 11:16:04 (1789139764)
[ 4152.799217] Lustre: 2351:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789139884/real 1789139884]  req@ffff93d35060a1c0 x1876044900070528/t0(0) o4->lustre-OST0000-osc-ffff93d3510b8000@192.168.203.139@tcp:6/4 lens 488/448 e 0 to 1 dl 1789139900 ref 2 fl Rpc:XQr/602/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4152.847146] Lustre: 2351:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 62 previous similar messages
[ 4200.863346] Lustre: lustre-OST0000-osc-ffff93d3510b8000: Connection to lustre-OST0000 (at 192.168.203.139@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4200.892639] Lustre: Skipped 15 previous similar messages
[ 4233.714275] Lustre: lustre-OST0000-osc-ffff93d3510b8000: Connection restored to 192.168.203.139@tcp (at 192.168.203.139@tcp)
[ 4233.726817] Lustre: Skipped 18 previous similar messages
[ 4310.012313] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 11:20:55 (1789140055)
[ 4357.664420] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 11:21:43 (1789140103)
[ 4360.815324] Lustre: Unmounted lustre-client
[ 4381.219724] Lustre: Mounted lustre-client - version 2.17.57_127_ged01a19
[ 4386.582116] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 11:22:12 (1789140132)
[ 4386.893693] LustreError: 56699:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4387.950990] LustreError: 56699:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4388.129617] Lustre: *** cfs_fail_loc=305, val=0***
[ 4410.801864] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 11:22:36 (1789140156)
[ 4411.251266] Lustre: Mounted lustre-client - version 2.17.57_127_ged01a19
[ 4413.274307] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4423.565687] Lustre: Unmounted lustre-client
[ 4431.514422] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 11:22:56 (1789140176)
[ 4525.401886] LustreError: lustre-MDT0000-mdc-ffff93d350276000: operation mds_reint to node 192.168.203.139@tcp failed: rc = -19
[ 4525.412399] LustreError: Skipped 1 previous similar message
[ 4545.503659] Lustre: 2352:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789140276/real 1789140276]  req@ffff93d348ff1e00 x1876044902743808/t0(0) o400->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 224/224 e 0 to 1 dl 1789140292 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4545.546308] Lustre: 2352:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 75 previous similar messages
[ 4545.553397] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 192.168.203.139@tcp) was lost; in progress operations using this service will fail
[ 4545.582321] Lustre: Evicted from MGS (at 192.168.203.139@tcp) after server handle changed from 0x42304a04bd0069b9 to 0x42304a04bd031cf9
[ 4707.234113] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 11:27:32 (1789140452)
[ 4712.952508] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4727.792110] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 192.168.203.139@tcp) was lost; in progress operations using this service will fail
[ 4727.824184] Lustre: Evicted from MGS (at 192.168.203.139@tcp) after server handle changed from 0x42304a04bd031cf9 to 0x42304a04bd07e83e
[ 4729.542918] LustreError: lustre-MDT0000-mdc-ffff93d350276000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4746.918047] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 11:28:12 (1789140492)
[ 4747.359607] Lustre: Mounted lustre-client - version 2.17.57_127_ged01a19
[ 4754.745863] LustreError: lustre-OST0000-osc-ffff93d350276000: operation ldlm_enqueue to node 192.168.203.139@tcp failed: rc = -107
[ 4754.769205] LustreError: lustre-OST0000-osc-ffff93d350276000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4766.363810] Lustre: Unmounted lustre-client
[ 4773.273369] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 11:28:39 (1789140519)
[ 4779.952454] LustreError: lustre-MDT0000-mdc-ffff93d350276000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4779.965866] LustreError: 61100:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4787.156747] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 11:28:53 (1789140533)
[ 4787.476193] LustreError: 61804:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4792.315866] LustreError: lustre-MDT0000-mdc-ffff93d350276000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4792.328993] LustreError: 61820:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4792.345601] LustreError: 61804:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=150
[ 4792.352163] LustreError: 61804:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting
[ 4792.399231] LustreError: 61826:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4793.509900] LustreError: 61833:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4793.522663] LustreError: 61833:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4795.916353] LustreError: 61855:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4795.928192] LustreError: 61855:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4800.432227] LustreError: 61900:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4800.444672] LustreError: 61900:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4810.701661] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 11:29:16 (1789140556)
[ 4817.920618] Lustre: lustre-OST0000-osc-ffff93d350276000: Connection to lustre-OST0000 (at 192.168.203.139@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4817.943769] Lustre: Skipped 16 previous similar messages
[ 4849.361835] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4851.005382] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4860.168566] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 11:30:06 (1789140606)
[ 4883.843633] Lustre: lustre-OST0000-osc-ffff93d350276000: Connection restored to 192.168.203.139@tcp (at 192.168.203.139@tcp)
[ 4883.852791] Lustre: Skipped 15 previous similar messages
[ 4896.490682] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4898.386328] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4907.385411] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 11:30:53 (1789140653)
[ 4967.698543] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4968.903886] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4976.351715] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 11:32:02 (1789140722)
[ 5011.908682] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5013.204603] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5019.110966] Lustre: Unmounted lustre-client
[ 5068.546525] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5071.033667] Lustre: Mounted lustre-client - version 2.17.57_127_ged01a19
[ 5077.170065] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 11:33:43 (1789140823)
[ 5079.356521] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5081.689132] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 11:33:47 (1789140827)
[ 5113.784148] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 11:34:19 (1789140859)
[ 5115.180913] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5117.010959] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 11:34:22 (1789140862)
[ 5118.170409] Lustre: *** cfs_fail_loc=805, val=0***
[ 5118.207781] Lustre: Mounted lustre-client - version 2.17.57_127_ged01a19
[ 5123.752638] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5144.032859] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 192.168.203.139@tcp) was lost; in progress operations using this service will fail
[ 5149.983195] Lustre: 2352:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789140881/real 1789140881]  req@ffff93d358d8a940 x1876044904801280/t0(0) o400->lustre-MDT0000-mdc-ffff93d361430800@192.168.203.139@tcp:12/10 lens 224/224 e 0 to 1 dl 1789140897 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5150.022089] Lustre: 2352:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
[ 5154.303856] Lustre: Evicted from MGS (at 192.168.203.139@tcp) after server handle changed from 0x42304a04bd07f477 to 0x42304a04bd07f79c
[ 5162.792790] Lustre: Unmounted lustre-client
[ 5169.524206] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 11:35:15 (1789140915)
[ 5196.273457] LustreError: 2348:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff93d3615712c0 x1876044904811648/t94489280516(94489280516) o101->lustre-MDT0000-mdc-ffff93d361430800@192.168.203.139@tcp:12/10 lens 664/608 e 0 to 0 dl 1789140959 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5202.619544] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5203.785407] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5211.690926] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 11:35:57 (1789140957)
[ 5212.889033] LustreError: lustre-OST0000-osc-ffff93d361430800: operation ost_write to node 192.168.203.139@tcp failed: rc = -107
[ 5212.893963] LustreError: Skipped 2 previous similar messages
[ 5212.904418] LustreError: lustre-OST0000-osc-ffff93d361430800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5212.915358] Lustre: 2349:0:(llite_lib.c:4366:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.139@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5212.938719] LustreError: 73002:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff93d361430800: namespace resource [0x240000400:0x3a62:0x0].0x0 (ffff93d348fbcb00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5221.747417] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 11:36:06 (1789140966)
[ 5223.201338] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5225.131929] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 11:36:10 (1789140970)
[ 5226.801944] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5228.348086] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 11:36:14 (1789140974)
[ 5229.586438] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5230.961426] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 11:36:17 (1789140977)
[ 5232.072828] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5233.585436] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 11:36:19 (1789140979)
[ 5234.727951] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5236.256725] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 11:36:22 (1789140982)
[ 5237.607168] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5239.240237] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 11:36:25 (1789140985)
[ 5240.555046] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5242.029912] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 11:36:28 (1789140988)
[ 5243.411105] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5244.839699] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 11:36:30 (1789140990)
[ 5246.147820] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5247.694367] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 11:36:33 (1789140993)
[ 5249.115897] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5251.063474] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 11:36:36 (1789140996)
[ 5252.731481] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5254.202414] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 11:36:40 (1789141000)
[ 5256.822493] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5258.474515] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 11:36:44 (1789141004)
[ 5286.989610] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 11:37:12 (1789141032)
[ 5315.005892] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 11:37:40 (1789141060)
[ 5315.455428] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5325.534611] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 11:37:51 (1789141071)
[ 5325.850771] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5334.761840] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 11:38:00 (1789141080)
[ 5335.156857] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5343.315200] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 11:38:09 (1789141089)
[ 5343.671263] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5351.806886] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 11:38:17 (1789141097)
[ 5352.127618] Lustre: *** cfs_fail_loc=510, val=3***
[ 5360.057563] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 11:38:26 (1789141106)
[ 5360.341430] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5370.238145] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 11:38:36 (1789141116)
[ 5370.571776] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5429.727238] Lustre: lustre-OST0001-osc-ffff93d361430800: Connection to lustre-OST0001 (at 192.168.203.139@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5429.735817] Lustre: Skipped 9 previous similar messages
[ 5435.728620] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 11:39:41 (1789141181)
[ 5436.139293] LustreError: 83206:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5439.006778] LustreError: lustre-MDT0000-mdc-ffff93d361430800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5439.030621] LustreError: 83220:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff93d361430800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff93d359fa9400) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5440.223137] LustreError: 83206:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5442.313923] LustreError: 83236:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5445.372673] LustreError: lustre-MDT0000-mdc-ffff93d361430800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5445.397363] LustreError: 83250:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff93d361430800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff93d358c14900) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5445.421418] LustreError: 83250:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5446.399140] LustreError: 83236:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5453.408669] LustreError: lustre-MDT0000-mdc-ffff93d361430800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5453.791876] LustreError: 83292:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5456.703489] LustreError: lustre-MDT0000-mdc-ffff93d361430800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5456.773751] LustreError: 83311:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5456.785505] LustreError: 83311:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5457.856614] LustreError: 83318:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5457.864864] LustreError: 83318:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5457.873231] LustreError: 83292:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5457.879437] LustreError: 83292:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff93d361430800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5460.122830] LustreError: 83340:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5460.133847] LustreError: 83340:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5463.634176] LustreError: 83367:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5466.528595] LustreError: lustre-MDT0000-mdc-ffff93d361430800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5466.550739] LustreError: 83381:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff93d361430800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff93d359fa9c00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5466.567627] LustreError: 83381:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5467.719140] LustreError: 83367:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5474.594896] LustreError: lustre-MDT0000-mdc-ffff93d361430800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5474.615962] LustreError: 83410:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff93d361430800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff93d3615f3600) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5474.650544] LustreError: 83410:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5474.668162] LustreError: 83415:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5474.674344] LustreError: 83415:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5474.941412] LustreError: 83423:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5477.732124] LustreError: lustre-MDT0000-mdc-ffff93d361430800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5479.023143] LustreError: 83423:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5481.975881] LustreError: lustre-MDT0000-mdc-ffff93d361430800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5483.146480] LustreError: 83476:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5483.160532] LustreError: 83476:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5483.216108] LustreError: 83451:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff93d361430800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5486.058725] Lustre: lustre-MDT0000-mdc-ffff93d361430800: Connection restored to 192.168.203.139@tcp (at 192.168.203.139@tcp)
[ 5486.070230] Lustre: Skipped 17 previous similar messages
[ 5491.014542] LustreError: 83522:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5491.021496] LustreError: 83522:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5494.557830] LustreError: lustre-MDT0000-mdc-ffff93d361430800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5495.111161] LustreError: 83522:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5495.122774] LustreError: 83522:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5495.130882] LustreError: 83522:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff93d361430800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5506.688774] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 11:40:52 (1789141252)
[ 5526.082828] LustreError: 70982:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5526.096482] LustreError: 70982:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5530.168323] LustreError: 70982:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5530.175582] LustreError: 70982:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5536.929829] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 11:41:22 (1789141282)
[ 5588.426572] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 11:42:14 (1789141334)
[ 5651.341932] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 11:43:17 (1789141397)
[ 5688.043585] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 11:43:53 (1789141433)
[ 5688.680150] Lustre: Mounted lustre-client - version 2.17.57_127_ged01a19
[ 5813.073559] Lustre: Unmounted lustre-client
[ 5819.359567] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 11:46:05 (1789141565)
[ 5820.012822] LustreError: 87849:0:(osc_request.c:3192:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5822.641057] LustreError: lustre-OST0000-osc-ffff93d361430800: operation ldlm_enqueue to node 192.168.203.139@tcp failed: rc = -107
[ 5822.649877] LustreError: Skipped 9 previous similar messages
[ 5822.665212] LustreError: lustre-OST0000-osc-ffff93d361430800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5822.674923] LustreError: 87932:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff93d361430800: namespace resource [0x240000400:0x3a8d:0x0].0x0 (ffff93d358c14800) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5822.693894] LustreError: 87932:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 2 previous similar messages
[ 5823.055183] LustreError: 87849:0:(osc_request.c:3192:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5823.065522] Lustre: 2352:0:(llite_lib.c:4366:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.139@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5830.555572] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 11:46:16 (1789141576)
[ 5887.967511] Lustre: 88516:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789141580/real 1789141580]  req@ffff93d351195a40 x1876044904983296/t0(0) o101->lustre-MDT0000-mdc-ffff93d361430800@192.168.203.139@tcp:12/10 lens 328/344 e 0 to 1 dl 1789141635 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 5887.995598] Lustre: 88516:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 15 previous similar messages
[ 5894.780147] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 11:47:20 (1789141640)
[ 5896.147648] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 5897.737671] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 11:47:23 (1789141643)
[ 5961.329719] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 5962.955665] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 11:48:28 (1789141708)
[ 6020.641849] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 11:49:26 (1789141766)
[ 6023.027687] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6024.481912] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 11:49:30 (1789141770)
[ 6025.654284] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6027.304898] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 11:49:33 (1789141773)
[ 6068.330987] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 11:50:14 (1789141814)
[ 6080.899051] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6106.015334] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 192.168.203.139@tcp) was lost; in progress operations using this service will fail
[ 6106.029867] LustreError: Skipped 2 previous similar messages
[ 6111.219359] Lustre: lustre-MDT0000-mdc-ffff93d361430800: Connection to lustre-MDT0000 (at 192.168.203.139@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6111.238548] Lustre: Skipped 16 previous similar messages
[ 6111.390731] Lustre: lustre-MDT0000-mdc-ffff93d361430800: Connection restored to 192.168.203.139@tcp (at 192.168.203.139@tcp)
[ 6111.401916] Lustre: Skipped 7 previous similar messages
[ 6116.453638] Lustre: Evicted from MGS (at 192.168.203.139@tcp) after server handle changed from 0x42304a04bd0802e7 to 0x42304a04bd081dd6
[ 6116.464152] Lustre: Skipped 2 previous similar messages
[ 6121.000704] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6122.667974] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6132.407384] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 11:51:18 (1789141878)
[ 6134.793149] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6136.353666] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 11:51:22 (1789141882)
[ 6147.572197] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 192.168.203.139@tcp) was lost; in progress operations using this service will fail
[ 6161.802631] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 11:51:47 (1789141907)
[ 6194.734382] Lustre: Evicted from MGS (at 192.168.203.139@tcp) after server handle changed from 0x42304a04bd081e77 to 0x42304a04bd08253f
[ 6194.751785] Lustre: Skipped 1 previous similar message
[ 6202.635600] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 11:52:28 (1789141948)
[ 6264.317198] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6268.247679] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6353.887326] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 192.168.203.139@tcp) was lost; in progress operations using this service will fail
[ 6353.901628] LustreError: Skipped 1 previous similar message
[ 6353.932436] Lustre: Evicted from MGS (at 192.168.203.139@tcp) after server handle changed from 0x42304a04bd08253f to 0x42304a04bd0830a6
[ 6369.202807] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6370.564461] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6410.131818] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6411.591651] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6555.394198] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 11:58:21 (1789142301)
[ 6566.300544] LustreError: lustre-OST0000-osc-ffff93d361430800: operation ost_setattr to node 192.168.203.139@tcp failed: rc = -19
[ 6566.317817] LustreError: Skipped 79 previous similar messages
[ 6617.431217] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6622.084657] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6887.603665] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 12:03:53 (1789142633)
[ 7024.620230] Lustre: lustre-MDT0000-mdc-ffff93d361430800: Connection to lustre-MDT0000 (at 192.168.203.139@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7024.641260] Lustre: Skipped 6 previous similar messages
[ 7039.982970] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 192.168.203.139@tcp) was lost; in progress operations using this service will fail
[ 7040.002040] LustreError: Skipped 1 previous similar message
[ 7040.014210] Lustre: Evicted from MGS (at 192.168.203.139@tcp) after server handle changed from 0x42304a04bd083465 to 0x42304a04bd17d75b
[ 7040.029360] Lustre: Skipped 1 previous similar message
[ 7040.040621] Lustre: MGC192.168.203.139@tcp: Connection restored to 192.168.203.139@tcp (at 192.168.203.139@tcp)
[ 7040.071165] Lustre: Skipped 9 previous similar messages
[ 7073.174487] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 12:06:59 (1789142819)
[ 7074.498504] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7076.272807] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 12:07:02 (1789142822)
[ 7078.359525] LustreError: lustre-MDT0000-mdc-ffff93d361430800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7078.384428] LustreError: 102210:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff93d361430800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff93d359e5cd00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 7078.403890] LustreError: 102210:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 7086.016252] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 12:07:11 (1789142831)
[ 7253.547400] LustreError: lustre-OST0000-osc-ffff93d361430800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7255.257989] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 12:10:01 (1789143001)
[ 7280.607140] Lustre: 2350:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789143007/real 1789143007]  req@ffff93d351021e00 x1876044922709632/t0(0) o4->lustre-OST0000-osc-ffff93d361430800@192.168.203.139@tcp:6/4 lens 4584/448 e 0 to 1 dl 1789143027 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7280.640596] Lustre: 2350:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 22 previous similar messages
[ 7298.368411] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 12:10:44 (1789143044)
[ 7299.875822] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7302.015538] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 12:10:47 (1789143047)
[ 7303.780790] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7305.795625] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 12:10:51 (1789143051)
[ 7343.310839] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 12:11:28 (1789143088)
[ 7381.993776] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 192.168.203.139@tcp) was lost; in progress operations using this service will fail
[ 7382.044053] Lustre: Evicted from MGS (at 192.168.203.139@tcp) after server handle changed from 0x42304a04bd17d75b to 0x42304a04bd17fe0c
[ 7382.052975] Lustre: Skipped 1 previous similar message
[ 7397.613241] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 12:12:23 (1789143143)
[ 7399.057116] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7400.571767] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 12:12:26 (1789143146)
[ 7402.025945] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7403.275933] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 12:12:29 (1789143149)
[ 7408.607121] Lustre: 2351:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789143118/real 1789143118]  req@ffff93d3480a43c0 x1876044922865792/t0(0) o400->lustre-MDT0000-mdc-ffff93d361430800@192.168.203.139@tcp:12/10 lens 224/224 e 0 to 1 dl 1789143155 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7412.564906] Lustre: Unmounted lustre-client
[ 7413.699542] Lustre: Mounted lustre-client - version 2.17.57_127_ged01a19
[ 7416.589573] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7462.769264] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2