[ 1941.104702] LustreError: lustre-OST0000-osc-ffff96fb44c29000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1941.162226] Lustre: 2364:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.143@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1947.301156] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 09:50:05 (1786629005)
[ 1948.923580] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1950.446985] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 09:50:08 (1786629008)
[ 1951.948900] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1953.658798] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 09:50:11 (1786629011)
[ 1975.786712] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 192.168.203.143@tcp) was lost; in progress operations using this service will fail
[ 1975.811645] Lustre: Evicted from MGS (at 192.168.203.143@tcp) after server handle changed from 0x82160fea94d26dd9 to 0x82160fea94d287c5
[ 2076.866241] LustreError: lustre-MDT0000-mdc-ffff96fb44c29000: operation ldlm_enqueue to node 192.168.203.143@tcp failed: rc = -19
[ 2076.871411] LustreError: Skipped 4 previous similar messages
[ 2093.536327] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 192.168.203.143@tcp) was lost; in progress operations using this service will fail
[ 2103.721502] Lustre: Evicted from MGS (at 192.168.203.143@tcp) after server handle changed from 0x82160fea94d287c5 to 0x82160fea94d4c74d
[ 2119.379476] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 09:52:58 (1786629178)
[ 2119.702875] Lustre: *** cfs_fail_loc=305, val=0***
[ 2119.709310] Lustre: Skipped 2 previous similar messages
[ 2165.219492] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 192.168.203.143@tcp) was lost; in progress operations using this service will fail
[ 2174.447299] Lustre: Evicted from MGS (at 192.168.203.143@tcp) after server handle changed from 0x82160fea94d4c74d to 0x82160fea94d4cb4b
[ 2189.674694] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2191.088339] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2198.602350] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 09:54:16 (1786629256)
[ 2210.287670] Lustre: lustre-MDT0000-mdc-ffff96fb44c29000: Connection to lustre-MDT0000 (at 192.168.203.143@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2210.294574] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 192.168.203.143@tcp) was lost; in progress operations using this service will fail
[ 2210.308563] Lustre: Skipped 12 previous similar messages
[ 2210.331327] Lustre: Evicted from MGS (at 192.168.203.143@tcp) after server handle changed from 0x82160fea94d4cb4b to 0x82160fea94d4d109
[ 2210.343766] Lustre: MGC192.168.203.143@tcp: Connection restored to 192.168.203.143@tcp (at 192.168.203.143@tcp)
[ 2210.355116] Lustre: Skipped 15 previous similar messages
[ 2210.381871] LustreError: lustre-MDT0000-mdc-ffff96fb44c29000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2232.267308] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 09:54:49 (1786629289)
[ 2263.094992] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 09:55:21 (1786629321)
[ 2276.585074] LustreError: lustre-MDT0000-mdc-ffff96fb44c29000: operation ldlm_enqueue to node 192.168.203.143@tcp failed: rc = -19
[ 2276.590109] LustreError: Skipped 1 previous similar message
[ 2297.327984] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 192.168.203.143@tcp) was lost; in progress operations using this service will fail
[ 2297.358239] Lustre: Evicted from MGS (at 192.168.203.143@tcp) after server handle changed from 0x82160fea94d4d109 to 0x82160fea94d52755
[ 2309.899890] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2311.721264] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2394.592150] Lustre: 2361:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786629438/real 1786629438]  req@ffff96fb60074380 x1873414373704448/t0(0) o400->MGC192.168.203.143@tcp@192.168.203.143@tcp:26/25 lens 224/224 e 0 to 1 dl 1786629454 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2394.635824] Lustre: 2361:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 10 previous similar messages
[ 2394.641423] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 192.168.203.143@tcp) was lost; in progress operations using this service will fail
[ 2404.842216] Lustre: Evicted from MGS (at 192.168.203.143@tcp) after server handle changed from 0x82160fea94d52755 to 0x82160fea94d7259f
[ 2420.606464] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2422.372927] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2487.598569] LustreError: lustre-MDT0000-mdc-ffff96fb44c29000: operation mds_reint to node 192.168.203.143@tcp failed: rc = -19
[ 2487.607642] LustreError: Skipped 1 previous similar message
[ 2506.228128] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 192.168.203.143@tcp) was lost; in progress operations using this service will fail
[ 2506.261886] Lustre: Evicted from MGS (at 192.168.203.143@tcp) after server handle changed from 0x82160fea94d7259f to 0x82160fea94d90fd0
[ 2507.504605] Lustre: 15922:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.143@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2532.882132] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2534.900062] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2562.924304] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 10:00:21 (1786629621)
[ 2587.120496] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 192.168.203.143@tcp) was lost; in progress operations using this service will fail
[ 2587.187622] Lustre: Evicted from MGS (at 192.168.203.143@tcp) after server handle changed from 0x82160fea94d90fd0 to 0x82160fea94d9f1eb
[ 2592.368318] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2622.027312] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2663.313248] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2702.106573] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2744.800406] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 192.168.203.143@tcp) was lost; in progress operations using this service will fail
[ 2744.811457] LustreError: Skipped 3 previous similar messages
[ 2755.003764] Lustre: Evicted from MGS (at 192.168.203.143@tcp) after server handle changed from 0x82160fea94dab264 to 0x82160fea94db016a
[ 2755.017915] Lustre: Skipped 3 previous similar messages
[ 2760.632806] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2788.339158] LustreError: lustre-MDT0000-mdc-ffff96fb44c29000: operation mds_close to node 192.168.203.143@tcp failed: rc = -19
[ 2788.348928] LustreError: Skipped 6 previous similar messages
[ 2816.075426] Lustre: MGC192.168.203.143@tcp: Connection restored to 192.168.203.143@tcp (at 192.168.203.143@tcp)
[ 2816.089530] Lustre: Skipped 16 previous similar messages
[ 2816.223596] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2848.898439] Lustre: lustre-MDT0000-mdc-ffff96fb44c29000: Connection to lustre-MDT0000 (at 192.168.203.143@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2848.919984] Lustre: Skipped 9 previous similar messages
[ 2868.404608] Lustre: 15922:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.143@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2907.357359] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 10:06:06 (1786629966)
[ 2954.771305] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 2956.940458] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3291.327749] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3294.405584] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3585.432808] LustreError: lustre-OST0000-osc-ffff96fb44c29000: operation ldlm_enqueue to node 192.168.203.143@tcp failed: rc = -19
[ 3585.446735] LustreError: Skipped 4 previous similar messages
[ 3585.454132] Lustre: lustre-OST0000-osc-ffff96fb44c29000: Connection to lustre-OST0000 (at 192.168.203.143@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3585.473556] Lustre: Skipped 2 previous similar messages
[ 3609.304700] Lustre: lustre-OST0000-osc-ffff96fb44c29000: Connection restored to 192.168.203.143@tcp (at 192.168.203.143@tcp)
[ 3609.319071] Lustre: Skipped 4 previous similar messages
[ 3626.782675] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3628.305096] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3879.994505] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 10:22:18 (1786630938)
[ 3897.313409] Lustre: 50576:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786630940/real 1786630940]  req@ffff96fb4aeef100 x1873414397248128/t0(0) o101->lustre-MDT0000-mdc-ffff96fb44c29000@192.168.203.143@tcp:12/10 lens 576/1152 e 0 to 1 dl 1786630956 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3897.364518] Lustre: 50576:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
[ 3904.738470] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 10:22:43 (1786630963)
[ 3928.934883] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 10:23:07 (1786630987)
[ 3952.291534] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 10:23:30 (1786631010)
[ 3952.825318] Lustre: Mounted lustre-client - version 2.17.57_1_gb6c95b3
[ 3984.864252] Lustre: 2364:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786631028/real 1786631028]  req@ffff96fb4af8a300 x1873414397266560/t0(0) o400->MGC192.168.203.143@tcp@192.168.203.143@tcp:26/25 lens 224/224 e 0 to 1 dl 1786631044 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3984.884569] Lustre: 2364:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 3984.893121] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 192.168.203.143@tcp) was lost; in progress operations using this service will fail
[ 3984.906183] LustreError: Skipped 2 previous similar messages
[ 3995.133609] Lustre: Evicted from MGS (at 192.168.203.143@tcp) after server handle changed from 0x82160fea94dc292a to 0x82160fea94f29cfc
[ 3995.149336] Lustre: Skipped 2 previous similar messages
[ 4001.079680] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4002.819748] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4005.120008] Lustre: Unmounted lustre-client
[ 4012.873577] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 10:24:30 (1786631070)
[ 4138.464155] Lustre: 2362:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786631182/real 1786631182]  req@ffff96fb4aeed500 x1873414397300608/t0(0) o4->lustre-OST0000-osc-ffff96fb44c29000@192.168.203.143@tcp:6/4 lens 488/448 e 0 to 1 dl 1786631198 ref 2 fl Rpc:XQr/602/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4138.502130] Lustre: 2362:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 68 previous similar messages
[ 4186.528259] Lustre: lustre-OST0000-osc-ffff96fb44c29000: Connection to lustre-OST0000 (at 192.168.203.143@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4186.559727] Lustre: Skipped 14 previous similar messages
[ 4218.365482] Lustre: lustre-OST0000-osc-ffff96fb44c29000: Connection restored to 192.168.203.143@tcp (at 192.168.203.143@tcp)
[ 4218.376626] Lustre: Skipped 16 previous similar messages
[ 4312.952443] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 10:29:30 (1786631370)
[ 4361.144762] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 10:30:19 (1786631419)
[ 4364.154857] Lustre: Unmounted lustre-client
[ 4386.831609] Lustre: Mounted lustre-client - version 2.17.57_1_gb6c95b3
[ 4394.283845] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 10:30:52 (1786631452)
[ 4394.877983] LustreError: 56499:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4395.946816] LustreError: 56499:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4396.161668] Lustre: *** cfs_fail_loc=305, val=0***
[ 4418.720758] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 10:31:16 (1786631476)
[ 4419.274852] Lustre: Mounted lustre-client - version 2.17.57_1_gb6c95b3
[ 4421.748419] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4432.089790] Lustre: Unmounted lustre-client
[ 4439.059203] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 10:31:37 (1786631497)
[ 4535.154076] LustreError: lustre-MDT0000-mdc-ffff96fb4a219000: operation mds_reint to node 192.168.203.143@tcp failed: rc = -19
[ 4553.700277] Lustre: 2361:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786631597/real 1786631597]  req@ffff96fb467c0700 x1873414399974016/t0(0) o400->MGC192.168.203.143@tcp@192.168.203.143@tcp:26/25 lens 224/224 e 0 to 1 dl 1786631613 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4553.749868] Lustre: 2361:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 78 previous similar messages
[ 4553.763882] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 192.168.203.143@tcp) was lost; in progress operations using this service will fail
[ 4563.941856] Lustre: Evicted from MGS (at 192.168.203.143@tcp) after server handle changed from 0x82160fea94f2b96c to 0x82160fea94f56eb9
[ 4748.441682] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 10:36:46 (1786631806)
[ 4753.606647] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4767.760890] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 192.168.203.143@tcp) was lost; in progress operations using this service will fail
[ 4767.814113] Lustre: Evicted from MGS (at 192.168.203.143@tcp) after server handle changed from 0x82160fea94f56eb9 to 0x82160fea94fa3917
[ 4767.869928] LustreError: lustre-MDT0000-mdc-ffff96fb4a219000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4786.609939] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 10:37:24 (1786631844)
[ 4787.148762] Lustre: Mounted lustre-client - version 2.17.57_1_gb6c95b3
[ 4794.750843] Lustre: lustre-OST0000-osc-ffff96fb4a219000: Connection to lustre-OST0000 (at 192.168.203.143@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4794.767066] Lustre: Skipped 11 previous similar messages
[ 4794.790065] LustreError: lustre-OST0000-osc-ffff96fb4a219000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4794.809949] LustreError: 60220:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff96fb4a219000: namespace resource [0x240000400:0x4363:0x0].0x0 (ffff96fb47c14900) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4806.871222] Lustre: Unmounted lustre-client
[ 4816.188638] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 10:37:53 (1786631873)
[ 4820.063294] Lustre: lustre-MDT0000-mdc-ffff96fb4a219000: Connection restored to 192.168.203.143@tcp (at 192.168.203.143@tcp)
[ 4820.084661] Lustre: Skipped 12 previous similar messages
[ 4823.420446] LustreError: lustre-MDT0000-mdc-ffff96fb4a219000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4823.461228] LustreError: 60891:0:(file.c:6165:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4830.639118] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 10:38:08 (1786631888)
[ 4830.937783] LustreError: 61594:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4836.320604] LustreError: 61594:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=0
[ 4836.331873] LustreError: 61594:0:(import.c:711:ptlrpc_connect_import_locked()) already connected
[ 4837.075870] LustreError: lustre-MDT0000-mdc-ffff96fb4a219000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4837.102205] LustreError: 61611:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4837.156330] LustreError: 61616:0:(file.c:6165:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4838.229955] LustreError: 61622:0:(file.c:6165:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4838.241959] LustreError: 61622:0:(file.c:6165:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4840.510811] LustreError: 61645:0:(file.c:6165:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4840.523911] LustreError: 61645:0:(file.c:6165:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4845.071168] LustreError: 61689:0:(file.c:6165:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4845.095642] LustreError: 61689:0:(file.c:6165:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4855.789525] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 10:38:34 (1786631914)
[ 4895.757896] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4897.348640] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4907.971190] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 10:39:26 (1786631966)
[ 4951.780530] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4954.272916] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4965.907242] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 10:40:24 (1786632024)
[ 5034.398342] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5035.749796] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5044.582650] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 10:41:42 (1786632102)
[ 5083.364510] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5084.982169] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5092.181243] Lustre: Unmounted lustre-client
[ 5143.262509] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5145.924385] Lustre: Mounted lustre-client - version 2.17.57_1_gb6c95b3
[ 5152.170397] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 10:43:30 (1786632210)
[ 5154.438311] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5156.068166] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 10:43:34 (1786632214)
[ 5189.914628] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 10:44:07 (1786632247)
[ 5191.560655] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5193.171706] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 10:44:11 (1786632251)
[ 5194.541215] Lustre: *** cfs_fail_loc=805, val=0***
[ 5194.613248] Lustre: Mounted lustre-client - version 2.17.57_1_gb6c95b3
[ 5199.831552] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5220.320196] Lustre: 2361:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786632264/real 1786632264]  req@ffff96fb45c53800 x1873414402033792/t0(0) o400->lustre-MDT0000-mdc-ffff96fb443e1800@192.168.203.143@tcp:12/10 lens 224/224 e 0 to 1 dl 1786632280 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5220.330920] Lustre: 2361:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5220.334510] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 192.168.203.143@tcp) was lost; in progress operations using this service will fail
[ 5220.355939] Lustre: Evicted from MGS (at 192.168.203.143@tcp) after server handle changed from 0x82160fea94fa43ac to 0x82160fea94fa46a0
[ 5229.050785] Lustre: Unmounted lustre-client
[ 5239.007450] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 10:44:56 (1786632296)
[ 5266.462592] LustreError: 2360:0:(client.c:3435:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff96fb441d1180 x1873414402045952/t90194313221(90194313221) o101->lustre-MDT0000-mdc-ffff96fb443e1800@192.168.203.143@tcp:12/10 lens 648/608 e 0 to 0 dl 1786632342 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5277.080710] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5278.878263] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5288.258915] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 10:45:46 (1786632346)
[ 5291.300848] LustreError: lustre-OST0000-osc-ffff96fb443e1800: operation ost_write to node 192.168.203.143@tcp failed: rc = -107
[ 5291.319313] LustreError: Skipped 3 previous similar messages
[ 5291.364714] LustreError: lustre-OST0000-osc-ffff96fb443e1800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5291.400879] Lustre: 2362:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.143@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5291.485591] LustreError: 72808:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff96fb443e1800: namespace resource [0x240000400:0x43c2:0x0].0x0 (ffff96fb47c69300) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5302.592794] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 10:45:59 (1786632359)
[ 5305.324951] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5307.376092] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 10:46:05 (1786632365)
[ 5309.803915] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5312.056591] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 10:46:09 (1786632369)
[ 5314.006550] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5316.092450] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 10:46:14 (1786632374)
[ 5318.240930] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5320.463624] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 10:46:18 (1786632378)
[ 5322.094746] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5324.172258] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 10:46:22 (1786632382)
[ 5325.672298] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5327.220864] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 10:46:25 (1786632385)
[ 5328.639451] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5330.498472] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 10:46:28 (1786632388)
[ 5332.041546] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5334.143834] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 10:46:32 (1786632392)
[ 5335.597956] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5337.305796] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 10:46:35 (1786632395)
[ 5338.761477] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5340.411759] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 10:46:38 (1786632398)
[ 5341.916377] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5343.542357] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 10:46:41 (1786632401)
[ 5346.140676] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5347.742657] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 10:46:46 (1786632406)
[ 5378.714987] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 10:47:17 (1786632437)
[ 5407.950839] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 10:47:46 (1786632466)
[ 5408.363233] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5419.110098] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 10:47:57 (1786632477)
[ 5419.774801] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5430.610780] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 10:48:08 (1786632488)
[ 5430.987737] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5441.236452] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 10:48:19 (1786632499)
[ 5441.451659] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5449.871646] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 10:48:28 (1786632508)
[ 5450.317471] Lustre: *** cfs_fail_loc=510, val=3***
[ 5459.273406] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 10:48:37 (1786632517)
[ 5459.654177] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5470.413869] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 10:48:48 (1786632528)
[ 5470.786386] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5530.592360] Lustre: lustre-OST0000-osc-ffff96fb443e1800: Connection to lustre-OST0000 (at 192.168.203.143@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5530.619805] Lustre: Skipped 15 previous similar messages
[ 5537.339839] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 10:49:55 (1786632595)
[ 5537.727257] LustreError: 83025:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5540.555435] LustreError: lustre-MDT0000-mdc-ffff96fb443e1800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5540.587433] LustreError: 83040:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff96fb443e1800: inode [0x20000a042:0x8:0x0] mdc close failed: rc = -108
[ 5540.624821] LustreError: 83040:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff96fb443e1800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff96fb47c69e00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5540.657800] LustreError: 83045:0:(file.c:6165:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5540.668780] Lustre: lustre-MDT0000-mdc-ffff96fb443e1800: Connection restored to 192.168.203.143@tcp (at 192.168.203.143@tcp)
[ 5540.671916] LustreError: 83045:0:(file.c:6165:ll_inode_revalidate_fini()) Skipped 20 previous similar messages
[ 5540.701373] Lustre: Skipped 14 previous similar messages
[ 5541.808158] LustreError: 83025:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5543.911233] LustreError: 83056:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5547.008219] LustreError: lustre-MDT0000-mdc-ffff96fb443e1800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5547.038327] LustreError: 83070:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff96fb443e1800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff96fb46766a00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5547.060928] LustreError: 83070:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5547.984265] LustreError: 83056:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5555.046806] LustreError: lustre-MDT0000-mdc-ffff96fb443e1800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5555.557152] LustreError: 83112:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5558.549874] LustreError: lustre-MDT0000-mdc-ffff96fb443e1800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5558.624682] LustreError: 83131:0:(file.c:6165:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5559.640176] LustreError: 83112:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5559.647082] LustreError: 83112:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff96fb443e1800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5559.663057] LustreError: 83112:0:(file.c:251:ll_close_inode_openhandle()) Skipped 1 previous similar message
[ 5560.878726] LustreError: 83149:0:(file.c:6165:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5560.892807] LustreError: 83149:0:(file.c:6165:ll_inode_revalidate_fini()) Skipped 12 previous similar messages
[ 5565.814177] LustreError: 83187:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5568.699097] LustreError: lustre-MDT0000-mdc-ffff96fb443e1800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5568.719242] LustreError: 83201:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff96fb443e1800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff96fb45ff8c00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5568.742372] LustreError: 83201:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5569.897317] LustreError: 83187:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5576.872021] LustreError: lustre-MDT0000-mdc-ffff96fb443e1800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5577.235635] LustreError: 83243:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5580.073247] LustreError: lustre-MDT0000-mdc-ffff96fb443e1800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5580.093563] LustreError: 83258:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff96fb443e1800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff96fb4af80600) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5580.127117] LustreError: 83258:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 2 previous similar messages
[ 5580.148564] LustreError: 83263:0:(file.c:6165:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5580.161710] LustreError: 83263:0:(file.c:6165:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5581.304752] LustreError: 83243:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5584.180311] LustreError: lustre-MDT0000-mdc-ffff96fb443e1800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5585.475139] LustreError: 83271:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff96fb443e1800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5593.338878] LustreError: 83342:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5593.359275] LustreError: 83342:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5596.615240] LustreError: lustre-MDT0000-mdc-ffff96fb443e1800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5596.675975] LustreError: 83365:0:(file.c:6165:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5596.687440] LustreError: 83365:0:(file.c:6165:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 5597.345955] LustreError: 83342:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5597.355850] LustreError: 83342:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5597.367608] LustreError: 83342:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff96fb443e1800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5608.343230] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 10:51:06 (1786632666)
[ 5628.536973] LustreError: 68263:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5628.546220] LustreError: 68263:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5632.616133] LustreError: 68263:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5632.627467] LustreError: 68263:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5639.199257] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 10:51:37 (1786632697)
[ 5689.851399] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 10:52:27 (1786632747)
[ 5754.709724] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 10:53:32 (1786632812)
[ 5789.865594] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 10:54:07 (1786632847)
[ 5790.283945] Lustre: Mounted lustre-client - version 2.17.57_1_gb6c95b3
[ 5913.922578] Lustre: Unmounted lustre-client
[ 5920.998426] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 10:56:19 (1786632979)
[ 5921.561150] LustreError: 87670:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5923.827293] LustreError: lustre-OST0000-osc-ffff96fb443e1800: operation ost_statfs to node 192.168.203.143@tcp failed: rc = -107
[ 5923.834948] LustreError: Skipped 9 previous similar messages
[ 5923.848229] LustreError: lustre-OST0000-osc-ffff96fb443e1800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5923.868594] LustreError: 87757:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff96fb443e1800: namespace resource [0x240000400:0x43ed:0x0].0x0 (ffff96fb47c69d00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5923.880267] LustreError: 87757:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5924.072132] LustreError: 87670:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5924.076687] Lustre: 2362:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.143@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5929.951418] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 10:56:28 (1786632988)
[ 5988.321058] Lustre: 88337:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786632992/real 1786632992]  req@ffff96fb49d2b800 x1873414402214784/t0(0) o101->lustre-MDT0000-mdc-ffff96fb443e1800@192.168.203.143@tcp:12/10 lens 328/344 e 0 to 1 dl 1786633047 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 5988.364845] Lustre: 88337:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 18 previous similar messages
[ 5994.994303] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 10:57:33 (1786633053)
[ 5996.455954] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 5997.992492] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 10:57:36 (1786633056)
[ 6062.243067] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6063.988204] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 10:58:42 (1786633122)
[ 6121.987534] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 10:59:40 (1786633180)
[ 6124.656766] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6127.196335] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 10:59:45 (1786633185)
[ 6128.858946] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6130.741445] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 10:59:48 (1786633188)
[ 6161.406316] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 11:00:19 (1786633219)
[ 6173.603832] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6198.240229] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 192.168.203.143@tcp) was lost; in progress operations using this service will fail
[ 6198.259868] LustreError: Skipped 2 previous similar messages
[ 6203.371937] Lustre: lustre-MDT0000-mdc-ffff96fb443e1800: Connection to lustre-MDT0000 (at 192.168.203.143@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6203.381632] Lustre: Skipped 16 previous similar messages
[ 6203.543412] Lustre: lustre-MDT0000-mdc-ffff96fb443e1800: Connection restored to 192.168.203.143@tcp (at 192.168.203.143@tcp)
[ 6203.555748] Lustre: Skipped 15 previous similar messages
[ 6207.532172] Lustre: Evicted from MGS (at 192.168.203.143@tcp) after server handle changed from 0x82160fea94fa5200 to 0x82160fea94fa6cc5
[ 6207.543697] Lustre: Skipped 2 previous similar messages
[ 6212.672973] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6214.049225] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6223.165222] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 11:01:21 (1786633281)
[ 6225.115339] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6226.638517] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 11:01:25 (1786633285)
[ 6238.187513] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 192.168.203.143@tcp) was lost; in progress operations using this service will fail
[ 6250.555889] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 11:01:48 (1786633308)
[ 6280.201824] Lustre: Evicted from MGS (at 192.168.203.143@tcp) after server handle changed from 0x82160fea94fa6e62 to 0x82160fea94fa70ed
[ 6280.221087] Lustre: Skipped 1 previous similar message
[ 6282.105170] Lustre: 68266:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.143@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6302.574706] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 11:02:40 (1786633360)
[ 6365.764590] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6368.634668] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6454.240382] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 192.168.203.143@tcp) was lost; in progress operations using this service will fail
[ 6454.265157] LustreError: Skipped 1 previous similar message
[ 6463.479415] Lustre: Evicted from MGS (at 192.168.203.143@tcp) after server handle changed from 0x82160fea94fa70ed to 0x82160fea94fa7f41
[ 6479.595736] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6481.458856] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6518.641821] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6520.046096] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6643.131141] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 11:08:21 (1786633701)
[ 6651.281391] LustreError: lustre-OST0000-osc-ffff96fb443e1800: operation ost_setattr to node 192.168.203.143@tcp failed: rc = -19
[ 6703.457315] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6707.355085] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6943.911284] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 11:13:22 (1786634002)
[ 7093.734086] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 192.168.203.143@tcp) was lost; in progress operations using this service will fail
[ 7093.734437] Lustre: lustre-MDT0000-mdc-ffff96fb443e1800: Connection to lustre-MDT0000 (at 192.168.203.143@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7093.734467] Lustre: Skipped 6 previous similar messages
[ 7093.761276] LustreError: Skipped 1 previous similar message
[ 7093.785313] Lustre: Evicted from MGS (at 192.168.203.143@tcp) after server handle changed from 0x82160fea94fa878a to 0x82160fea95099dca
[ 7093.798547] Lustre: Skipped 1 previous similar message
[ 7093.807499] Lustre: MGC192.168.203.143@tcp: Connection restored to 192.168.203.143@tcp (at 192.168.203.143@tcp)
[ 7093.820770] Lustre: Skipped 9 previous similar messages
[ 7127.291070] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 11:16:25 (1786634185)
[ 7128.851744] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7130.702356] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 11:16:28 (1786634188)
[ 7132.515839] LustreError: lustre-MDT0000-mdc-ffff96fb443e1800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7132.541115] LustreError: 102021:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff96fb443e1800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff96fb54c6cd00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 7132.560317] LustreError: 102021:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 7140.897466] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 11:16:38 (1786634198)
[ 7307.497428] LustreError: lustre-OST0000-osc-ffff96fb443e1800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7309.070548] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 11:19:27 (1786634367)
[ 7334.368160] Lustre: 2364:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786634373/real 1786634373]  req@ffff96fb5d16d180 x1873414419275520/t0(0) o4->lustre-OST0000-osc-ffff96fb443e1800@192.168.203.143@tcp:6/4 lens 4584/448 e 0 to 1 dl 1786634393 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7334.401645] Lustre: 2364:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 28 previous similar messages
[ 7351.400061] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 11:20:09 (1786634409)
[ 7352.757361] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7354.173385] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 11:20:12 (1786634412)
[ 7355.749843] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7357.595484] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 11:20:15 (1786634415)
[ 7391.436156] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 11:20:49 (1786634449)
[ 7431.658781] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 192.168.203.143@tcp) was lost; in progress operations using this service will fail
[ 7431.681685] Lustre: Evicted from MGS (at 192.168.203.143@tcp) after server handle changed from 0x82160fea95099dca to 0x82160fea9509d775
[ 7431.696584] Lustre: Skipped 1 previous similar message
[ 7444.984729] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 11:21:43 (1786634503)
[ 7446.619436] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7448.323107] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 11:21:46 (1786634506)
[ 7449.891737] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7451.773953] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 11:21:49 (1786634509)
[ 7466.235224] Lustre: Unmounted lustre-client
[ 7467.858236] Lustre: Mounted lustre-client - version 2.17.57_1_gb6c95b3
[ 7470.287967] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7489.312226] Lustre: 2363:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786634532/real 1786634532]  req@ffff96fb495f2300 x1873414419462016/t0(0) o400->MGC192.168.203.143@tcp@192.168.203.143@tcp:26/25 lens 224/224 e 0 to 1 dl 1786634548 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7499.141990] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2