[ 1846.802716] LustreError: lustre-OST0000-osc-ffff948351d1c000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1846.830319] Lustre: 2349:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.143@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1853.780555] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 07:53:00 (1788263580)
[ 1856.324333] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1858.219306] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 07:53:05 (1788263585)
[ 1859.821107] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1861.886188] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 07:53:08 (1788263588)
[ 1883.619306] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 192.168.203.143@tcp) was lost; in progress operations using this service will fail
[ 1883.649122] Lustre: Evicted from MGS (at 192.168.203.143@tcp) after server handle changed from 0x6d5d58f56dc67b32 to 0x6d5d58f56dc694c3
[ 1986.190726] LustreError: lustre-MDT0000-mdc-ffff948351d1c000: operation mds_close to node 192.168.203.143@tcp failed: rc = -19
[ 1986.205781] LustreError: Skipped 3 previous similar messages
[ 2005.472683] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 192.168.203.143@tcp) was lost; in progress operations using this service will fail
[ 2015.668863] Lustre: Evicted from MGS (at 192.168.203.143@tcp) after server handle changed from 0x6d5d58f56dc694c3 to 0x6d5d58f56dc81f40
[ 2030.387072] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 07:55:57 (1788263757)
[ 2031.004220] Lustre: *** cfs_fail_loc=305, val=0***
[ 2031.006409] Lustre: Skipped 4 previous similar messages
[ 2069.920345] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 192.168.203.143@tcp) was lost; in progress operations using this service will fail
[ 2080.240929] Lustre: Evicted from MGS (at 192.168.203.143@tcp) after server handle changed from 0x6d5d58f56dc81f40 to 0x6d5d58f56dc82249
[ 2096.485463] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2098.066437] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2107.093710] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 07:57:13 (1788263833)
[ 2126.312128] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 192.168.203.143@tcp) was lost; in progress operations using this service will fail
[ 2126.373672] Lustre: Evicted from MGS (at 192.168.203.143@tcp) after server handle changed from 0x6d5d58f56dc82249 to 0x6d5d58f56dc8283f
[ 2126.473665] LustreError: lustre-MDT0000-mdc-ffff948351d1c000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2153.193160] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 07:57:59 (1788263879)
[ 2165.771375] LustreError: lustre-OST0000-osc-ffff948351d1c000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2185.329532] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 07:58:31 (1788263911)
[ 2199.576119] LustreError: lustre-MDT0000-mdc-ffff948351d1c000: operation ldlm_enqueue to node 192.168.203.143@tcp failed: rc = -19
[ 2219.489585] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 192.168.203.143@tcp) was lost; in progress operations using this service will fail
[ 2219.526181] Lustre: Evicted from MGS (at 192.168.203.143@tcp) after server handle changed from 0x6d5d58f56dc8283f to 0x6d5d58f56dc86b75
[ 2247.019656] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2249.791535] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2336.229990] Lustre: 2349:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788264048/real 1788264048]  req@ffff948350fd2a00 x1875128379152768/t0(0) o400->MGC192.168.203.143@tcp@192.168.203.143@tcp:26/25 lens 224/224 e 0 to 1 dl 1788264064 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2336.288387] Lustre: 2349:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 13 previous similar messages
[ 2336.347673] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 192.168.203.143@tcp) was lost; in progress operations using this service will fail
[ 2346.553419] Lustre: Evicted from MGS (at 192.168.203.143@tcp) after server handle changed from 0x6d5d58f56dc86b75 to 0x6d5d58f56dc9f7ce
[ 2364.953758] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2367.598694] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2433.654211] LustreError: lustre-MDT0000-mdc-ffff948351d1c000: operation mds_close to node 192.168.203.143@tcp failed: rc = -19
[ 2433.682294] LustreError: Skipped 2 previous similar messages
[ 2453.473199] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 192.168.203.143@tcp) was lost; in progress operations using this service will fail
[ 2463.674738] Lustre: Evicted from MGS (at 192.168.203.143@tcp) after server handle changed from 0x6d5d58f56dc9f7ce to 0x6d5d58f56dcb7efc
[ 2463.733950] Lustre: MGC192.168.203.143@tcp: Connection restored to 192.168.203.143@tcp (at 192.168.203.143@tcp)
[ 2463.752430] Lustre: Skipped 14 previous similar messages
[ 2483.673984] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2486.804140] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2519.508109] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 08:04:05 (1788264245)
[ 2526.628615] Lustre: lustre-MDT0000-mdc-ffff948351d1c000: Connection to lustre-MDT0000 (at 192.168.203.143@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2526.652405] Lustre: Skipped 9 previous similar messages
[ 2544.609677] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 192.168.203.143@tcp) was lost; in progress operations using this service will fail
[ 2553.844376] Lustre: Evicted from MGS (at 192.168.203.143@tcp) after server handle changed from 0x6d5d58f56dcb7efc to 0x6d5d58f56dcc37e2
[ 2557.972904] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2599.500082] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2638.824761] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2693.356575] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2717.737794] LustreError: lustre-MDT0000-mdc-ffff948351d1c000: operation mds_reint to node 192.168.203.143@tcp failed: rc = -19
[ 2717.756555] LustreError: Skipped 4 previous similar messages
[ 2737.120434] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 192.168.203.143@tcp) was lost; in progress operations using this service will fail
[ 2737.137582] LustreError: Skipped 3 previous similar messages
[ 2747.380369] Lustre: Evicted from MGS (at 192.168.203.143@tcp) after server handle changed from 0x6d5d58f56dccbc80 to 0x6d5d58f56dcd413a
[ 2747.389960] Lustre: Skipped 3 previous similar messages
[ 2748.791619] Lustre: 15908:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.143@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2760.342995] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2819.787268] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2912.169276] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 08:10:38 (1788264638)
[ 2972.879852] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 2975.761457] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3258.482818] LustreError: lustre-OST0000-osc-ffff948351d1c000: operation ldlm_enqueue to node 192.168.203.143@tcp failed: rc = -107
[ 3258.488878] LustreError: Skipped 4 previous similar messages
[ 3258.494467] Lustre: lustre-OST0000-osc-ffff948351d1c000: Connection to lustre-OST0000 (at 192.168.203.143@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3258.508484] Lustre: Skipped 6 previous similar messages
[ 3281.295750] Lustre: lustre-OST0000-osc-ffff948351d1c000: Connection restored to 192.168.203.143@tcp (at 192.168.203.143@tcp)
[ 3281.312993] Lustre: Skipped 15 previous similar messages
[ 3299.648558] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3302.162813] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3631.034641] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3633.358148] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3884.120585] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 08:26:51 (1788265611)
[ 3900.896295] Lustre: 50563:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788265613/real 1788265613]  req@ffff9483508cfb80 x1875128399084288/t0(0) o101->lustre-MDT0000-mdc-ffff948351d1c000@192.168.203.143@tcp:12/10 lens 576/1152 e 0 to 1 dl 1788265629 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3900.929326] Lustre: 50563:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
[ 3900.934482] Lustre: lustre-MDT0000-mdc-ffff948351d1c000: Connection to lustre-MDT0000 (at 192.168.203.143@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3900.943650] Lustre: Skipped 1 previous similar message
[ 3900.973934] Lustre: lustre-MDT0000-mdc-ffff948351d1c000: Connection restored to 192.168.203.143@tcp (at 192.168.203.143@tcp)
[ 3900.980548] Lustre: Skipped 1 previous similar message
[ 3908.900410] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 08:27:15 (1788265635)
[ 3933.665166] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 08:27:40 (1788265660)
[ 3959.828278] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 08:28:06 (1788265686)
[ 3960.414972] Lustre: Mounted lustre-client - version 2.17.57_103_ga9ee299
[ 3991.008398] Lustre: 2350:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788265703/real 1788265703]  req@ffff948350e01500 x1875128399103360/t0(0) o400->lustre-MDT0000-mdc-ffff948351d1c000@192.168.203.143@tcp:12/10 lens 224/224 e 0 to 1 dl 1788265719 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3991.065580] Lustre: 2350:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 3991.067699] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 192.168.203.143@tcp) was lost; in progress operations using this service will fail
[ 3991.099796] LustreError: Skipped 2 previous similar messages
[ 4000.753297] Lustre: Evicted from MGS (at 192.168.203.143@tcp) after server handle changed from 0x6d5d58f56dce72a8 to 0x6d5d58f56de153c6
[ 4000.771722] Lustre: Skipped 2 previous similar messages
[ 4008.595548] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4010.258396] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4012.313932] Lustre: Unmounted lustre-client
[ 4017.813788] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 08:29:04 (1788265744)
[ 4156.384612] Lustre: 2351:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788265868/real 1788265868]  req@ffff9483508ce680 x1875128399137152/t0(0) o4->lustre-OST0000-osc-ffff948351d1c000@192.168.203.143@tcp:6/4 lens 488/448 e 0 to 1 dl 1788265884 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4156.427644] Lustre: 2351:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 74 previous similar messages
[ 4314.227586] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 08:34:00 (1788266040)
[ 4363.126392] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 08:34:50 (1788266090)
[ 4366.082494] Lustre: Unmounted lustre-client
[ 4389.181742] Lustre: Mounted lustre-client - version 2.17.57_103_ga9ee299
[ 4394.680296] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 08:35:21 (1788266121)
[ 4395.092887] LustreError: 56490:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4396.145596] LustreError: 56490:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4396.268720] Lustre: *** cfs_fail_loc=305, val=0***
[ 4419.248281] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 08:35:46 (1788266146)
[ 4419.830481] Lustre: Mounted lustre-client - version 2.17.57_103_ga9ee299
[ 4422.995799] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4433.310157] Lustre: Unmounted lustre-client
[ 4441.596498] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 08:36:08 (1788266168)
[ 4545.621032] LustreError: lustre-MDT0000-mdc-ffff9483495c0800: operation mds_reint to node 192.168.203.143@tcp failed: rc = -19
[ 4545.631924] LustreError: Skipped 1 previous similar message
[ 4545.636286] Lustre: lustre-MDT0000-mdc-ffff9483495c0800: Connection to lustre-MDT0000 (at 192.168.203.143@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4545.657232] Lustre: Skipped 23 previous similar messages
[ 4564.448495] Lustre: 2351:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788266276/real 1788266276]  req@ffff948350fbc000 x1875128401808000/t0(0) o400->MGC192.168.203.143@tcp@192.168.203.143@tcp:26/25 lens 224/224 e 0 to 1 dl 1788266292 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4564.491989] Lustre: 2351:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 73 previous similar messages
[ 4564.501210] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 192.168.203.143@tcp) was lost; in progress operations using this service will fail
[ 4574.696749] Lustre: Evicted from MGS (at 192.168.203.143@tcp) after server handle changed from 0x6d5d58f56de169de to 0x6d5d58f56de41d41
[ 4574.730038] Lustre: MGC192.168.203.143@tcp: Connection restored to 192.168.203.143@tcp (at 192.168.203.143@tcp)
[ 4574.751597] Lustre: Skipped 24 previous similar messages
[ 4749.552397] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 08:41:16 (1788266476)
[ 4754.677985] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4767.210295] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 192.168.203.143@tcp) was lost; in progress operations using this service will fail
[ 4767.232299] Lustre: Evicted from MGS (at 192.168.203.143@tcp) after server handle changed from 0x6d5d58f56de41d41 to 0x6d5d58f56de8e920
[ 4768.819525] LustreError: lustre-MDT0000-mdc-ffff9483495c0800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4783.951754] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 08:41:51 (1788266511)
[ 4784.395753] Lustre: Mounted lustre-client - version 2.17.57_103_ga9ee299
[ 4791.672819] LustreError: lustre-OST0000-osc-ffff9483495c0800: operation ldlm_enqueue to node 192.168.203.143@tcp failed: rc = -107
[ 4791.693989] LustreError: lustre-OST0000-osc-ffff9483495c0800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4791.715938] LustreError: 60212:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff9483495c0800: namespace resource [0x240000400:0x3903:0x0].0x0 (ffff948350739300) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4803.046561] Lustre: Unmounted lustre-client
[ 4809.754127] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 08:42:16 (1788266536)
[ 4816.325742] LustreError: lustre-MDT0000-mdc-ffff9483495c0800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4816.340348] LustreError: 60885:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4823.294350] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 08:42:30 (1788266550)
[ 4823.582280] LustreError: 61589:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4827.570099] LustreError: lustre-MDT0000-mdc-ffff9483495c0800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4827.583299] LustreError: 61606:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4827.597885] LustreError: 61589:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=1002
[ 4827.611483] LustreError: 61589:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting
[ 4827.632276] LustreError: 61611:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4828.696731] LustreError: 61617:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4828.715770] LustreError: 61617:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4830.985725] LustreError: 61640:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4831.000191] LustreError: 61640:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4835.506890] LustreError: 61685:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4835.514881] LustreError: 61685:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4845.150444] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 08:42:52 (1788266572)
[ 4883.829738] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4885.486711] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4896.155594] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 08:43:42 (1788266622)
[ 4931.565634] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4933.049855] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4942.458429] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 08:44:29 (1788266669)
[ 5001.123037] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5002.343838] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5009.735528] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 08:45:36 (1788266736)
[ 5044.377157] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5045.779463] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5050.873318] Lustre: Unmounted lustre-client
[ 5095.500394] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5097.626974] Lustre: Mounted lustre-client - version 2.17.57_103_ga9ee299
[ 5102.447989] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 08:47:09 (1788266829)
[ 5104.308480] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5105.495892] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 08:47:12 (1788266832)
[ 5127.551777] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 08:47:34 (1788266854)
[ 5128.746887] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5130.008359] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 08:47:37 (1788266857)
[ 5131.126471] Lustre: *** cfs_fail_loc=805, val=0***
[ 5131.171488] Lustre: Mounted lustre-client - version 2.17.57_103_ga9ee299
[ 5135.202814] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5156.833641] Lustre: lustre-MDT0000-mdc-ffff948351d72800: Connection to lustre-MDT0000 (at 192.168.203.143@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5156.839001] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 192.168.203.143@tcp) was lost; in progress operations using this service will fail
[ 5156.848947] Lustre: Skipped 14 previous similar messages
[ 5156.866541] Lustre: Evicted from MGS (at 192.168.203.143@tcp) after server handle changed from 0x6d5d58f56de8f560 to 0x6d5d58f56de8f8f5
[ 5160.906423] Lustre: Unmounted lustre-client
[ 5166.292669] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 08:48:13 (1788266893)
[ 5167.008613] Lustre: 2349:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788266879/real 1788266879]  req@ffff948344869180 x1875128403865728/t0(0) o400->lustre-MDT0000-mdc-ffff948351d71000@192.168.203.143@tcp:12/10 lens 224/224 e 0 to 1 dl 1788266895 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5167.033382] Lustre: 2349:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
[ 5187.525632] Lustre: MGC192.168.203.143@tcp: Connection restored to 192.168.203.143@tcp (at 192.168.203.143@tcp)
[ 5187.533406] Lustre: Skipped 16 previous similar messages
[ 5193.737961] LustreError: 2348:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff94834660d180 x1875128403874176/t90194313220(90194313220) o101->lustre-MDT0000-mdc-ffff948351d71000@192.168.203.143@tcp:12/10 lens 664/608 e 0 to 0 dl 1788266937 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5197.807384] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5199.111276] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5205.821387] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 08:48:52 (1788266932)
[ 5206.978226] LustreError: lustre-OST0000-osc-ffff948351d71000: operation ost_write to node 192.168.203.143@tcp failed: rc = -107
[ 5206.987088] LustreError: Skipped 2 previous similar messages
[ 5207.005696] LustreError: lustre-OST0000-osc-ffff948351d71000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5207.026909] Lustre: 2351:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.143@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5207.061753] LustreError: 72791:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff948351d71000: namespace resource [0x240000400:0x3962:0x0].0x0 (ffff948346677400) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5213.164846] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 08:49:00 (1788266940)
[ 5214.122069] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5215.337133] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 08:49:02 (1788266942)
[ 5216.382830] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5217.509078] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 08:49:04 (1788266944)
[ 5218.614222] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5219.667963] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 08:49:07 (1788266947)
[ 5220.850825] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5222.083693] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 08:49:09 (1788266949)
[ 5222.911564] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5223.879653] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 08:49:11 (1788266951)
[ 5224.714772] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5225.647686] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 08:49:13 (1788266953)
[ 5226.496506] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5227.577805] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 08:49:14 (1788266954)
[ 5228.513853] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5229.581513] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 08:49:16 (1788266956)
[ 5230.428990] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5231.520839] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 08:49:18 (1788266958)
[ 5232.504607] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5233.827476] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 08:49:21 (1788266961)
[ 5234.954749] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5236.351498] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 08:49:23 (1788266963)
[ 5237.990895] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5239.075950] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 08:49:26 (1788266966)
[ 5261.745624] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 08:49:49 (1788266989)
[ 5288.516423] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 08:50:15 (1788267015)
[ 5288.907371] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5297.712790] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 08:50:24 (1788267024)
[ 5297.987363] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5306.196373] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 08:50:33 (1788267033)
[ 5306.374868] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5312.501146] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 08:50:39 (1788267039)
[ 5312.776880] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5318.953540] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 08:50:46 (1788267046)
[ 5319.359285] Lustre: *** cfs_fail_loc=510, val=3***
[ 5325.949606] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 08:50:53 (1788267053)
[ 5326.278318] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5334.651154] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 08:51:02 (1788267062)
[ 5335.004330] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5398.777988] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 08:52:06 (1788267126)
[ 5398.975905] LustreError: 82990:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5401.498112] LustreError: lustre-MDT0000-mdc-ffff948351d71000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5401.512515] LustreError: 83004:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff948351d71000: inode [0x20000a042:0x8:0x0] mdc close failed: rc = -108
[ 5401.531760] LustreError: 83004:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff948351d71000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff948344ae2f00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5401.536391] LustreError: 83009:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5401.557302] LustreError: 83009:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 14 previous similar messages
[ 5403.040262] LustreError: 82990:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5405.114879] LustreError: 83020:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5407.822073] LustreError: lustre-MDT0000-mdc-ffff948351d71000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5407.839959] LustreError: 83034:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff948351d71000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff948350739c00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5407.853956] LustreError: 83034:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5409.176132] LustreError: 83020:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5415.800468] LustreError: lustre-MDT0000-mdc-ffff948351d71000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5416.083798] LustreError: 83076:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5418.640398] LustreError: lustre-MDT0000-mdc-ffff948351d71000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5418.676949] LustreError: 83096:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5420.153693] LustreError: 83076:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5420.161432] LustreError: 83076:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff948351d71000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5420.173660] LustreError: 83076:0:(file.c:251:ll_close_inode_openhandle()) Skipped 2 previous similar messages
[ 5425.362950] LustreError: 83151:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5427.936551] LustreError: lustre-MDT0000-mdc-ffff948351d71000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5427.952906] LustreError: 83166:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff948351d71000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff948344ae2500) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5427.963918] LustreError: 83166:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5429.432115] LustreError: 83151:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5436.157304] LustreError: lustre-MDT0000-mdc-ffff948351d71000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5436.448833] LustreError: 83207:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5439.043950] LustreError: lustre-MDT0000-mdc-ffff948351d71000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5439.061505] LustreError: 83222:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff948351d71000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff948350739700) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5439.075555] LustreError: 83222:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5440.512135] LustreError: 83207:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5443.271938] LustreError: lustre-MDT0000-mdc-ffff948351d71000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5444.680896] LustreError: 83235:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff948351d71000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5452.063867] LustreError: 83307:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5452.071640] LustreError: 83307:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5455.007158] LustreError: lustre-MDT0000-mdc-ffff948351d71000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5455.075738] LustreError: 83329:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5455.082716] LustreError: 83329:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 53 previous similar messages
[ 5456.136248] LustreError: 83307:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5456.148934] LustreError: 83307:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5456.160765] LustreError: 83307:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff948351d71000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5463.605891] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 08:53:10 (1788267190)
[ 5481.030506] LustreError: 68245:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5481.038783] LustreError: 68245:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5485.105701] LustreError: 68245:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5485.112590] LustreError: 68245:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5489.466216] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 08:53:36 (1788267216)
[ 5537.852537] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 08:54:25 (1788267265)
[ 5598.861895] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 08:55:26 (1788267326)
[ 5627.828971] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 08:55:55 (1788267355)
[ 5628.082738] Lustre: Mounted lustre-client - version 2.17.57_103_ga9ee299
[ 5750.341433] Lustre: Unmounted lustre-client
[ 5755.022898] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 08:58:02 (1788267482)
[ 5755.503195] LustreError: 87628:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5757.244094] LustreError: lustre-OST0000-osc-ffff948351d71000: operation ost_statfs to node 192.168.203.143@tcp failed: rc = -107
[ 5757.250244] LustreError: Skipped 9 previous similar messages
[ 5757.253405] Lustre: lustre-OST0000-osc-ffff948351d71000: Connection to lustre-OST0000 (at 192.168.203.143@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5757.262149] Lustre: Skipped 16 previous similar messages
[ 5757.273102] LustreError: lustre-OST0000-osc-ffff948351d71000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5757.287734] LustreError: 87715:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff948351d71000: namespace resource [0x240000400:0x398d:0x0].0x0 (ffff948350739600) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5757.307220] LustreError: 87715:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5757.488117] LustreError: 87628:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5757.491703] Lustre: 2352:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.143@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5761.867220] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 08:58:09 (1788267489)
[ 5820.384229] Lustre: 88295:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788267492/real 1788267492]  req@ffff948344868a80 x1875128404037504/t0(0) o101->lustre-MDT0000-mdc-ffff948351d71000@192.168.203.143@tcp:12/10 lens 328/344 e 0 to 1 dl 1788267548 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 5820.402978] Lustre: 88295:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
[ 5820.418683] Lustre: lustre-MDT0000-mdc-ffff948351d71000: Connection restored to 192.168.203.143@tcp (at 192.168.203.143@tcp)
[ 5820.425068] Lustre: Skipped 16 previous similar messages
[ 5824.688972] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 08:59:12 (1788267552)
[ 5825.620253] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 5826.771662] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 08:59:14 (1788267554)
[ 5888.222367] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 5889.244400] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 09:00:16 (1788267616)
[ 5947.251992] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 09:01:14 (1788267674)
[ 5948.726257] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 5949.811556] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 09:01:17 (1788267677)
[ 5950.698565] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 5951.731616] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 09:01:19 (1788267679)
[ 5971.998275] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 09:01:39 (1788267699)
[ 5979.095328] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6002.667464] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 192.168.203.143@tcp) was lost; in progress operations using this service will fail
[ 6002.677801] LustreError: Skipped 2 previous similar messages
[ 6002.696391] Lustre: Evicted from MGS (at 192.168.203.143@tcp) after server handle changed from 0x6d5d58f56de903b4 to 0x6d5d58f56de91dae
[ 6002.707033] Lustre: Skipped 2 previous similar messages
[ 6008.033300] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6008.829915] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6015.411634] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 09:02:22 (1788267742)
[ 6017.238298] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6018.299937] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 09:02:25 (1788267745)
[ 6033.014195] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 09:02:40 (1788267760)
[ 6056.652757] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 09:03:04 (1788267784)
[ 6086.642174] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6087.907804] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6174.043120] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6174.697261] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6197.791170] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6198.487164] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6287.822598] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 09:06:55 (1788268015)
[ 6319.815256] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6321.653239] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6512.231179] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 09:10:39 (1788268239)
[ 6560.742092] Lustre: lustre-MDT0000-mdc-ffff948351d71000: Connection to lustre-MDT0000 (at 192.168.203.143@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6560.748633] Lustre: Skipped 10 previous similar messages
[ 6565.858745] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 192.168.203.143@tcp) was lost; in progress operations using this service will fail
[ 6565.863919] LustreError: Skipped 4 previous similar messages
[ 6565.903779] Lustre: Evicted from MGS (at 192.168.203.143@tcp) after server handle changed from 0x6d5d58f56de936b3 to 0x6d5d58f56df8d8bb
[ 6565.908882] Lustre: Skipped 4 previous similar messages
[ 6565.912383] Lustre: MGC192.168.203.143@tcp: Connection restored to 192.168.203.143@tcp (at 192.168.203.143@tcp)
[ 6565.915904] Lustre: Skipped 12 previous similar messages
[ 6584.571670] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 09:11:52 (1788268312)
[ 6585.098564] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 6585.722753] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 09:11:53 (1788268313)
[ 6586.409810] LustreError: lustre-MDT0000-mdc-ffff948351d71000: operation ldlm_enqueue to node 192.168.203.143@tcp failed: rc = -107
[ 6586.414347] LustreError: Skipped 84 previous similar messages
[ 6586.419061] LustreError: lustre-MDT0000-mdc-ffff948351d71000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6589.363416] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 09:11:57 (1788268317)
[ 6754.786941] LustreError: lustre-OST0000-osc-ffff948351d71000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6755.446724] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 09:14:43 (1788268483)
[ 6777.312108] Lustre: 2351:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788268485/real 1788268485]  req@ffff948347393480 x1875128421723648/t0(0) o4->lustre-OST0000-osc-ffff948351d71000@192.168.203.143@tcp:6/4 lens 4584/448 e 0 to 1 dl 1788268505 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 6777.322261] Lustre: 2351:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 13 previous similar messages
[ 6789.224380] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 09:15:17 (1788268517)
[ 6789.766825] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 6790.361832] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 09:15:18 (1788268518)
[ 6790.902313] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 6791.481966] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 09:15:19 (1788268519)
[ 6816.412470] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 09:15:44 (1788268544)
[ 6849.912221] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 09:16:17 (1788268577)
[ 6850.452768] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 6851.098826] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 09:16:18 (1788268578)
[ 6851.703353] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 6852.314403] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 09:16:20 (1788268580)
[ 6855.443190] Lustre: Unmounted lustre-client
[ 6855.883461] Lustre: Mounted lustre-client - version 2.17.57_103_ga9ee299
[ 6856.897914] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6873.439632] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2