[ 1897.166649] LustreError: lustre-OST0000-osc-ffff93cf0946a800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1897.205522] Lustre: 2359:0:(llite_lib.c:4366:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.107@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1904.493488] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 06:58:26 (1789124306)
[ 1906.382927] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1908.055756] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 06:58:30 (1789124310)
[ 1909.787993] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1911.532700] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 06:58:33 (1789124313)
[ 1915.217361] LustreError: lustre-MDT0000-mdc-ffff93cf0946a800: operation mds_close to node 192.168.203.107@tcp failed: rc = -19
[ 1915.227341] LustreError: Skipped 1 previous similar message
[ 1933.793531] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 192.168.203.107@tcp) was lost; in progress operations using this service will fail
[ 1943.991581] Lustre: Evicted from MGS (at 192.168.203.107@tcp) after server handle changed from 0x7e943b74488c0152 to 0x7e943b74488c1d44
[ 2037.505834] LustreError: lustre-MDT0000-mdc-ffff93cf0946a800: operation ldlm_enqueue to node 192.168.203.107@tcp failed: rc = -19
[ 2056.674380] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 192.168.203.107@tcp) was lost; in progress operations using this service will fail
[ 2065.913365] Lustre: Evicted from MGS (at 192.168.203.107@tcp) after server handle changed from 0x7e943b74488c1d44 to 0x7e943b74488dbd0e
[ 2082.169389] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 07:01:24 (1789124484)
[ 2082.588806] Lustre: *** cfs_fail_loc=305, val=0***
[ 2082.600221] Lustre: Skipped 2 previous similar messages
[ 2122.208346] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 192.168.203.107@tcp) was lost; in progress operations using this service will fail
[ 2132.462255] Lustre: Evicted from MGS (at 192.168.203.107@tcp) after server handle changed from 0x7e943b74488dbd0e to 0x7e943b74488dc0f0
[ 2145.144613] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2146.727847] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2154.587530] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 07:02:36 (1789124556)
[ 2168.301449] Lustre: lustre-MDT0000-mdc-ffff93cf0946a800: Connection to lustre-MDT0000 (at 192.168.203.107@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2168.320723] Lustre: Skipped 12 previous similar messages
[ 2168.325844] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 192.168.203.107@tcp) was lost; in progress operations using this service will fail
[ 2168.343857] Lustre: Evicted from MGS (at 192.168.203.107@tcp) after server handle changed from 0x7e943b74488dc0f0 to 0x7e943b74488dc5f1
[ 2168.360464] Lustre: MGC192.168.203.107@tcp: Connection restored to 192.168.203.107@tcp (at 192.168.203.107@tcp)
[ 2168.368533] Lustre: Skipped 15 previous similar messages
[ 2168.410416] LustreError: lustre-MDT0000-mdc-ffff93cf0946a800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2189.108453] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 07:03:11 (1789124591)
[ 2222.024703] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 07:03:43 (1789124623)
[ 2235.223717] LustreError: lustre-MDT0000-mdc-ffff93cf0946a800: operation mds_reint to node 192.168.203.107@tcp failed: rc = -19
[ 2255.336845] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 192.168.203.107@tcp) was lost; in progress operations using this service will fail
[ 2255.372206] Lustre: Evicted from MGS (at 192.168.203.107@tcp) after server handle changed from 0x7e943b74488dc5f1 to 0x7e943b74488e0be3
[ 2273.553898] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2276.361593] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2362.848276] Lustre: 2361:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789124750/real 1789124750]  req@ffff93cf09395e00 x1876030899286656/t0(0) o400->MGC192.168.203.107@tcp@192.168.203.107@tcp:26/25 lens 224/224 e 0 to 1 dl 1789124766 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2362.879956] Lustre: 2361:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 12 previous similar messages
[ 2362.889851] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 192.168.203.107@tcp) was lost; in progress operations using this service will fail
[ 2362.909458] Lustre: Evicted from MGS (at 192.168.203.107@tcp) after server handle changed from 0x7e943b74488e0be3 to 0x7e943b74488fd211
[ 2364.411731] Lustre: 16130:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.107@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2399.663850] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2401.580036] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2466.733478] LustreError: lustre-MDT0000-mdc-ffff93cf0946a800: operation mds_reint to node 192.168.203.107@tcp failed: rc = -19
[ 2466.745941] LustreError: Skipped 4 previous similar messages
[ 2486.691038] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 192.168.203.107@tcp) was lost; in progress operations using this service will fail
[ 2486.732316] Lustre: Evicted from MGS (at 192.168.203.107@tcp) after server handle changed from 0x7e943b74488fd211 to 0x7e943b744891841f
[ 2512.709856] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2514.981918] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2543.859983] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 07:09:05 (1789124945)
[ 2566.624226] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 192.168.203.107@tcp) was lost; in progress operations using this service will fail
[ 2575.258602] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2575.860727] Lustre: Evicted from MGS (at 192.168.203.107@tcp) after server handle changed from 0x7e943b744891841f to 0x7e943b7448925236
[ 2607.871435] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2643.620910] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2685.055280] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2729.440319] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 192.168.203.107@tcp) was lost; in progress operations using this service will fail
[ 2729.457913] LustreError: Skipped 3 previous similar messages
[ 2739.628894] Lustre: Evicted from MGS (at 192.168.203.107@tcp) after server handle changed from 0x7e943b7448929009 to 0x7e943b7448932976
[ 2739.644062] Lustre: Skipped 3 previous similar messages
[ 2746.595102] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2774.630173] LustreError: lustre-MDT0000-mdc-ffff93cf0946a800: operation mds_close to node 192.168.203.107@tcp failed: rc = -19
[ 2774.635978] Lustre: lustre-MDT0000-mdc-ffff93cf0946a800: Connection to lustre-MDT0000 (at 192.168.203.107@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2774.644731] LustreError: Skipped 10 previous similar messages
[ 2774.663047] Lustre: Skipped 8 previous similar messages
[ 2794.683925] Lustre: MGC192.168.203.107@tcp: Connection restored to 192.168.203.107@tcp (at 192.168.203.107@tcp)
[ 2794.696783] Lustre: Skipped 16 previous similar messages
[ 2802.557899] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2900.911466] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 07:15:02 (1789125302)
[ 2961.745738] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 2964.470974] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3293.996931] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3296.248166] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3578.701918] LustreError: lustre-OST0000-osc-ffff93cf0946a800: operation ldlm_enqueue to node 192.168.203.107@tcp failed: rc = -107
[ 3578.717048] LustreError: Skipped 4 previous similar messages
[ 3578.721090] Lustre: lustre-OST0000-osc-ffff93cf0946a800: Connection to lustre-OST0000 (at 192.168.203.107@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3578.730351] Lustre: Skipped 3 previous similar messages
[ 3615.450424] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3617.232139] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3873.097365] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 07:31:15 (1789126275)
[ 3890.144566] Lustre: 50773:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789126277/real 1789126277]  req@ffff93cf2111a580 x1876030920365184/t0(0) o101->lustre-MDT0000-mdc-ffff93cf0946a800@192.168.203.107@tcp:12/10 lens 576/1152 e 0 to 1 dl 1789126293 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3890.194807] Lustre: 50773:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
[ 3890.252720] Lustre: lustre-MDT0000-mdc-ffff93cf0946a800: Connection restored to 192.168.203.107@tcp (at 192.168.203.107@tcp)
[ 3890.264267] Lustre: Skipped 3 previous similar messages
[ 3898.489957] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 07:31:40 (1789126300)
[ 3926.417810] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 07:32:07 (1789126327)
[ 3951.582947] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 07:32:33 (1789126353)
[ 3952.069628] Lustre: Mounted lustre-client - version 2.17.57_139_gd84534a
[ 3982.816156] Lustre: 2360:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789126370/real 1789126370]  req@ffff93cf21118000 x1876030920384768/t0(0) o400->lustre-MDT0000-mdc-ffff93cf1082c000@192.168.203.107@tcp:12/10 lens 224/224 e 0 to 1 dl 1789126386 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3982.816252] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 192.168.203.107@tcp) was lost; in progress operations using this service will fail
[ 3982.861475] Lustre: 2360:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 3982.895962] LustreError: Skipped 2 previous similar messages
[ 3993.063717] Lustre: Evicted from MGS (at 192.168.203.107@tcp) after server handle changed from 0x7e943b7448944c19 to 0x7e943b7448a88621
[ 3993.080760] Lustre: Skipped 2 previous similar messages
[ 3999.918885] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4001.504711] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4003.972899] Lustre: Unmounted lustre-client
[ 4009.437811] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 07:33:31 (1789126411)
[ 4146.658307] Lustre: 2361:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789126533/real 1789126533]  req@ffff93cf07fc4780 x1876030920416896/t0(0) o4->lustre-OST0000-osc-ffff93cf0946a800@192.168.203.107@tcp:6/4 lens 488/448 e 0 to 1 dl 1789126549 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4146.707400] Lustre: 2361:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 59 previous similar messages
[ 4193.768908] Lustre: lustre-OST0000-osc-ffff93cf0946a800: Connection to lustre-OST0000 (at 192.168.203.107@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4193.791265] Lustre: Skipped 15 previous similar messages
[ 4305.243333] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 07:38:27 (1789126707)
[ 4357.234359] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 07:39:18 (1789126758)
[ 4360.702487] Lustre: Unmounted lustre-client
[ 4379.288022] Lustre: Mounted lustre-client - version 2.17.57_139_gd84534a
[ 4384.845203] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 07:39:47 (1789126787)
[ 4385.063347] LustreError: 56696:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4386.114292] LustreError: 56696:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4386.273570] Lustre: *** cfs_fail_loc=305, val=0***
[ 4408.672244] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 07:40:10 (1789126810)
[ 4409.057652] Lustre: Mounted lustre-client - version 2.17.57_139_gd84534a
[ 4410.996051] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4421.322426] Lustre: Unmounted lustre-client
[ 4427.026106] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 07:40:29 (1789126829)
[ 4517.487130] LustreError: lustre-MDT0000-mdc-ffff93cf0289e000: operation mds_reint to node 192.168.203.107@tcp failed: rc = -19
[ 4537.322074] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 192.168.203.107@tcp) was lost; in progress operations using this service will fail
[ 4537.341588] Lustre: Evicted from MGS (at 192.168.203.107@tcp) after server handle changed from 0x7e943b7448a89deb to 0x7e943b7448ab540a
[ 4537.354514] Lustre: MGC192.168.203.107@tcp: Connection restored to 192.168.203.107@tcp (at 192.168.203.107@tcp)
[ 4537.362616] Lustre: Skipped 23 previous similar messages
[ 4539.303930] Lustre: 56094:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.107@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 4705.822691] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 07:45:07 (1789127107)
[ 4710.130671] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4725.760872] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 192.168.203.107@tcp) was lost; in progress operations using this service will fail
[ 4725.810969] Lustre: Evicted from MGS (at 192.168.203.107@tcp) after server handle changed from 0x7e943b7448ab540a to 0x7e943b7448b01a78
[ 4725.835955] LustreError: lustre-MDT0000-mdc-ffff93cf0289e000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4737.158898] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 07:45:39 (1789127139)
[ 4737.572305] Lustre: Mounted lustre-client - version 2.17.57_139_gd84534a
[ 4744.804141] LustreError: lustre-OST0000-osc-ffff93cf0289e000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4744.817112] LustreError: 60419:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff93cf0289e000: namespace resource [0x240000400:0x3c23:0x0].0x0 (ffff93cf059c1400) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4754.914896] Lustre: 2358:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789127142/real 1789127142]  req@ffff93cf045d1680 x1876030925070848/t0(0) o103->lustre-OST0000-osc-ffff93cf090b8800@192.168.203.107@tcp:17/18 lens 328/224 e 0 to 1 dl 1789127158 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295
[ 4754.938628] Lustre: 2358:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 78 previous similar messages
[ 4756.927702] Lustre: Unmounted lustre-client
[ 4762.361472] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 07:46:04 (1789127164)
[ 4768.660179] LustreError: lustre-MDT0000-mdc-ffff93cf0289e000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4768.681532] LustreError: 61089:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4775.161847] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 07:46:17 (1789127177)
[ 4775.345782] LustreError: 61793:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4780.266815] LustreError: lustre-MDT0000-mdc-ffff93cf0289e000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4780.289825] LustreError: 61810:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4780.302366] LustreError: 61793:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=59
[ 4780.314076] LustreError: 61793:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting
[ 4780.345154] LustreError: 61815:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4781.411390] LustreError: 61822:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4781.423753] LustreError: 61822:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4783.713598] LustreError: 61844:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4783.733377] LustreError: 61844:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4788.274341] LustreError: 61889:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4788.284760] LustreError: 61889:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4798.219199] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 07:46:40 (1789127200)
[ 4806.130245] Lustre: lustre-OST0000-osc-ffff93cf0289e000: Connection to lustre-OST0000 (at 192.168.203.107@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4806.140908] Lustre: Skipped 17 previous similar messages
[ 4838.034222] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4839.500914] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4849.051472] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 07:47:31 (1789127251)
[ 4884.236428] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4885.386354] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4895.112509] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 07:48:16 (1789127296)
[ 4955.884539] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4957.659793] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4964.956315] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 07:49:27 (1789127367)
[ 4999.318938] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5000.901658] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5006.374809] Lustre: Unmounted lustre-client
[ 5050.578894] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5052.947895] Lustre: Mounted lustre-client - version 2.17.57_139_gd84534a
[ 5059.763839] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 07:51:01 (1789127461)
[ 5062.433611] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5064.498537] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 07:51:06 (1789127466)
[ 5095.892607] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 07:51:38 (1789127498)
[ 5097.102912] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5098.363243] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 07:51:40 (1789127500)
[ 5099.431598] Lustre: *** cfs_fail_loc=805, val=0***
[ 5099.477743] Lustre: Mounted lustre-client - version 2.17.57_139_gd84534a
[ 5104.029393] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5125.088563] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 192.168.203.107@tcp) was lost; in progress operations using this service will fail
[ 5135.334416] Lustre: Evicted from MGS (at 192.168.203.107@tcp) after server handle changed from 0x7e943b7448b024a4 to 0x7e943b7448b02855
[ 5135.359273] LustreError: 2357:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff93cf18072000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff93cf05a87500) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5135.387723] LustreError: 2357:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5138.153544] Lustre: Unmounted lustre-client
[ 5147.310087] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 07:52:29 (1789127549)
[ 5172.140406] Lustre: MGC192.168.203.107@tcp: Connection restored to 192.168.203.107@tcp (at 192.168.203.107@tcp)
[ 5172.150485] Lustre: Skipped 15 previous similar messages
[ 5174.249280] LustreError: 2357:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff93cf045d1680 x1876030925150848/t90194313221(90194313221) o101->lustre-MDT0000-mdc-ffff93cf02ce4000@192.168.203.107@tcp:12/10 lens 648/608 e 0 to 0 dl 1789127593 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5185.350500] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5186.719089] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5194.672049] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 07:53:16 (1789127596)
[ 5196.034512] LustreError: lustre-OST0000-osc-ffff93cf02ce4000: operation ost_write to node 192.168.203.107@tcp failed: rc = -107
[ 5196.041888] LustreError: Skipped 3 previous similar messages
[ 5196.056289] LustreError: lustre-OST0000-osc-ffff93cf02ce4000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5196.071167] Lustre: 2358:0:(llite_lib.c:4366:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.107@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5196.120128] LustreError: 72997:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff93cf02ce4000: namespace resource [0x240000400:0x3c82:0x0].0x0 (ffff93cf058d2900) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5196.145301] LustreError: 72997:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5203.652654] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 07:53:25 (1789127605)
[ 5205.143259] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5206.622330] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 07:53:28 (1789127608)
[ 5208.186594] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5209.892915] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 07:53:31 (1789127611)
[ 5211.179152] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5212.731858] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 07:53:34 (1789127614)
[ 5214.184959] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5215.459281] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 07:53:37 (1789127617)
[ 5216.670261] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5218.247597] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 07:53:40 (1789127620)
[ 5219.683981] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5221.349671] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 07:53:43 (1789127623)
[ 5222.771679] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5224.221507] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 07:53:46 (1789127626)
[ 5225.644068] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5227.496558] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 07:53:49 (1789127629)
[ 5228.950711] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5230.797755] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 07:53:52 (1789127632)
[ 5232.143873] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5233.854748] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 07:53:55 (1789127635)
[ 5235.252783] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5237.260941] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 07:53:59 (1789127639)
[ 5239.377840] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5241.361645] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 07:54:03 (1789127643)
[ 5271.568871] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 07:54:33 (1789127673)
[ 5299.994499] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 07:55:02 (1789127702)
[ 5300.445129] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5309.931687] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 07:55:11 (1789127711)
[ 5310.293262] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5321.143192] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 07:55:22 (1789127722)
[ 5321.446959] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5329.909642] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 07:55:31 (1789127731)
[ 5330.128670] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5338.752638] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 07:55:40 (1789127740)
[ 5339.152776] Lustre: *** cfs_fail_loc=510, val=3***
[ 5347.739127] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 07:55:49 (1789127749)
[ 5348.137642] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5358.034451] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 07:56:00 (1789127760)
[ 5358.318795] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5416.928216] Lustre: 2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789127765/real 1789127765]  req@ffff93cf045d1a40 x1876030925194752/t0(0) o3->lustre-OST0000-osc-ffff93cf02ce4000@192.168.203.107@tcp:6/4 lens 488/4536 e 0 to 1 dl 1789127820 ref 2 fl Rpc:RXQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 5416.967752] Lustre: 2359:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 15 previous similar messages
[ 5416.976957] Lustre: lustre-OST0000-osc-ffff93cf02ce4000: Connection to lustre-OST0000 (at 192.168.203.107@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5416.996758] Lustre: Skipped 8 previous similar messages
[ 5423.222427] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 07:57:05 (1789127825)
[ 5423.713990] LustreError: 83197:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5426.518119] LustreError: lustre-MDT0000-mdc-ffff93cf02ce4000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5426.537071] LustreError: 83212:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff93cf02ce4000: inode [0x20000a042:0x8:0x0] mdc close failed: rc = -108
[ 5426.553480] LustreError: 83212:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff93cf02ce4000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff93cf08511900) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5427.792154] LustreError: 83197:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5429.881416] LustreError: 83227:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5432.614545] LustreError: lustre-MDT0000-mdc-ffff93cf02ce4000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5432.626830] LustreError: 83241:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff93cf02ce4000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff93cf08511100) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5432.649026] LustreError: 83241:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5433.968182] LustreError: 83227:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5440.888498] LustreError: lustre-MDT0000-mdc-ffff93cf02ce4000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5440.900810] LustreError: 83270:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff93cf02ce4000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff93cf0596ca00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5440.918185] LustreError: 83270:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5441.351634] LustreError: 83283:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5444.199470] LustreError: lustre-MDT0000-mdc-ffff93cf02ce4000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5444.247589] LustreError: 83302:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5444.258912] LustreError: 83302:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 20 previous similar messages
[ 5445.347211] LustreError: 83308:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5445.356394] LustreError: 83308:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5445.432107] LustreError: 83283:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5445.440244] LustreError: 83283:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff93cf02ce4000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5447.622555] LustreError: 83331:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5447.627528] LustreError: 83331:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5451.140638] LustreError: 83357:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5453.862591] LustreError: lustre-MDT0000-mdc-ffff93cf02ce4000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5455.224486] LustreError: 83357:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5462.140851] LustreError: lustre-MDT0000-mdc-ffff93cf02ce4000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5462.160102] LustreError: 83400:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff93cf02ce4000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff93cf0596cd00) refcount nonzero (0) after lock cleanup; forcing cleanup.
[ 5462.184637] LustreError: 83400:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 2 previous similar messages
[ 5462.527900] LustreError: 83413:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5465.381886] LustreError: lustre-MDT0000-mdc-ffff93cf02ce4000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5466.608167] LustreError: 83413:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5469.474645] LustreError: lustre-MDT0000-mdc-ffff93cf02ce4000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5469.523167] LustreError: 83460:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5469.530260] LustreError: 83460:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5470.768891] LustreError: 83440:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff93cf02ce4000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5478.357188] LustreError: 83511:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5478.364920] LustreError: 83511:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5481.974823] LustreError: lustre-MDT0000-mdc-ffff93cf02ce4000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5482.048473] LustreError: 83534:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5482.061462] LustreError: 83534:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5482.448960] LustreError: 83511:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5482.460250] LustreError: 83511:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5482.473630] LustreError: 83511:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff93cf02ce4000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5492.200932] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 07:58:14 (1789127894)
[ 5511.027150] LustreError: 79950:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5511.035499] LustreError: 79950:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5515.097364] LustreError: 79950:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5515.107204] LustreError: 79950:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5520.382828] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 07:58:42 (1789127922)
[ 5568.759766] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 07:59:30 (1789127970)
[ 5631.219879] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 08:00:33 (1789128033)
[ 5664.974216] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 08:01:06 (1789128066)
[ 5665.502434] Lustre: Mounted lustre-client - version 2.17.57_139_gd84534a
[ 5789.030165] Lustre: Unmounted lustre-client
[ 5795.561989] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 08:03:17 (1789128197)
[ 5796.058117] LustreError: 87851:0:(osc_request.c:3192:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5798.368150] LustreError: lustre-OST0000-osc-ffff93cf02ce4000: operation ost_statfs to node 192.168.203.107@tcp failed: rc = -107
[ 5798.376516] LustreError: Skipped 9 previous similar messages
[ 5798.384982] LustreError: lustre-OST0000-osc-ffff93cf02ce4000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5798.395059] LustreError: 87938:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff93cf02ce4000: namespace resource [0x240000400:0x3cad:0x0].0x0 (ffff93cf08511c00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5798.406061] LustreError: 87938:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 2 previous similar messages
[ 5798.568150] LustreError: 87851:0:(osc_request.c:3192:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5798.573742] Lustre: 2360:0:(llite_lib.c:4366:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.107@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5798.586815] Lustre: lustre-OST0000-osc-ffff93cf02ce4000: Connection restored to 192.168.203.107@tcp (at 192.168.203.107@tcp)
[ 5798.596879] Lustre: Skipped 15 previous similar messages
[ 5804.386634] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 08:03:26 (1789128206)
[ 5874.247729] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 08:04:36 (1789128276)
[ 5875.790397] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 5877.988516] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 08:04:39 (1789128279)
[ 5940.821384] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 5942.570684] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 08:05:44 (1789128344)
[ 6001.308966] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 08:06:43 (1789128403)
[ 6003.267594] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6004.900996] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 08:06:46 (1789128406)
[ 6006.226996] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6007.790656] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 08:06:49 (1789128409)
[ 6039.772573] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 08:07:21 (1789128441)
[ 6050.816304] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6077.408445] Lustre: 2361:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789128464/real 1789128464]  req@ffff93cf07eeb840 x1876030925350144/t0(0) o400->MGC192.168.203.107@tcp@192.168.203.107@tcp:26/25 lens 224/224 e 0 to 1 dl 1789128480 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6077.461964] Lustre: 2361:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
[ 6077.469702] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 192.168.203.107@tcp) was lost; in progress operations using this service will fail
[ 6077.486453] LustreError: Skipped 2 previous similar messages
[ 6077.513567] Lustre: lustre-MDT0000-mdc-ffff93cf02ce4000: Connection to lustre-MDT0000 (at 192.168.203.107@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6077.520773] Lustre: Evicted from MGS (at 192.168.203.107@tcp) after server handle changed from 0x7e943b7448b032ce to 0x7e943b7448b04b78
[ 6077.529736] Lustre: Skipped 16 previous similar messages
[ 6077.551115] Lustre: Skipped 2 previous similar messages
[ 6091.669664] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6093.047448] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6101.481626] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 08:08:23 (1789128503)
[ 6103.887148] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6105.519298] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 08:08:27 (1789128507)
[ 6118.375335] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 192.168.203.107@tcp) was lost; in progress operations using this service will fail
[ 6118.396963] Lustre: Evicted from MGS (at 192.168.203.107@tcp) after server handle changed from 0x7e943b7448b04b78 to 0x7e943b7448b0501e
[ 6130.688662] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 08:08:52 (1789128532)
[ 6161.388151] Lustre: 68457:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.107@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6187.467328] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 08:09:49 (1789128589)
[ 6242.716693] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6245.810523] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6332.192227] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 192.168.203.107@tcp) was lost; in progress operations using this service will fail
[ 6332.206237] LustreError: Skipped 1 previous similar message
[ 6332.235050] Lustre: Evicted from MGS (at 192.168.203.107@tcp) after server handle changed from 0x7e943b7448b0522b to 0x7e943b7448b0607f
[ 6332.244360] Lustre: Skipped 1 previous similar message
[ 6333.756168] Lustre: 68457:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.107@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6356.594041] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6358.794437] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6396.755756] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6398.329449] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6543.375277] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 08:15:44 (1789128944)
[ 6553.175740] LustreError: lustre-OST0000-osc-ffff93cf02ce4000: operation ost_setattr to node 192.168.203.107@tcp failed: rc = -19
[ 6553.191952] LustreError: Skipped 7 previous similar messages
[ 6611.381735] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6615.411376] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6867.780368] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 08:21:09 (1789129269)
[ 7003.116126] Lustre: lustre-MDT0000-mdc-ffff93cf02ce4000: Connection to lustre-MDT0000 (at 192.168.203.107@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7003.143536] Lustre: Skipped 6 previous similar messages
[ 7013.364883] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 192.168.203.107@tcp) was lost; in progress operations using this service will fail
[ 7013.378470] LustreError: Skipped 1 previous similar message
[ 7013.404533] Lustre: Evicted from MGS (at 192.168.203.107@tcp) after server handle changed from 0x7e943b7448b064ae to 0x7e943b7448bfeaa1
[ 7013.415985] Lustre: Skipped 1 previous similar message
[ 7013.423728] Lustre: MGC192.168.203.107@tcp: Connection restored to 192.168.203.107@tcp (at 192.168.203.107@tcp)
[ 7013.435359] Lustre: Skipped 13 previous similar messages
[ 7053.646545] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 08:24:15 (1789129455)
[ 7055.435479] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7058.054097] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 08:24:19 (1789129459)
[ 7060.666633] LustreError: lustre-MDT0000-mdc-ffff93cf02ce4000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7069.575749] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 08:24:31 (1789129471)
[ 7237.038725] LustreError: lustre-OST0000-osc-ffff93cf02ce4000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7238.830865] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 08:27:20 (1789129640)
[ 7264.736643] Lustre: 2361:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789129647/real 1789129647]  req@ffff93cf083f4780 x1876030942926848/t0(0) o4->lustre-OST0000-osc-ffff93cf02ce4000@192.168.203.107@tcp:6/4 lens 4584/448 e 0 to 1 dl 1789129667 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7264.770469] Lustre: 2361:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 16 previous similar messages
[ 7264.802322] Lustre: lustre-OST0000-osc-ffff93cf02ce4000: Connection restored to 192.168.203.107@tcp (at 192.168.203.107@tcp)
[ 7264.811692] Lustre: Skipped 2 previous similar messages
[ 7281.547969] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 08:28:03 (1789129683)
[ 7282.805068] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7284.140401] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 08:28:06 (1789129686)
[ 7285.664712] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7287.044303] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 08:28:09 (1789129689)
[ 7322.189606] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 08:28:43 (1789129723)
[ 7359.987925] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 192.168.203.107@tcp) was lost; in progress operations using this service will fail
[ 7360.051026] Lustre: Evicted from MGS (at 192.168.203.107@tcp) after server handle changed from 0x7e943b7448bfeaa1 to 0x7e943b7448c0125c
[ 7360.077155] Lustre: Skipped 1 previous similar message
[ 7379.548348] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 08:29:41 (1789129781)
[ 7381.057177] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7382.637870] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 08:29:44 (1789129784)
[ 7384.169659] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7385.879097] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 08:29:47 (1789129787)
[ 7397.397929] Lustre: Unmounted lustre-client
[ 7399.335624] Lustre: Mounted lustre-client - version 2.17.57_139_gd84534a
[ 7401.829080] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7430.129591] Lustre: MGC192.168.203.107@tcp: Connection restored to 192.168.203.107@tcp (at 192.168.203.107@tcp)
[ 7430.142281] Lustre: Skipped 2 previous similar messages
[ 7440.682845] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2