[ 1442.982043] LustreError: lustre-OST0000-osc-ffff8800a8d1b800: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1442.990031] Lustre: 2009:0:(llite_lib.c:4135:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.104@tcp:/lustre/fid: [0x240000403:0xb:0x0]// may get corrupted (rc -108) [ 1445.103560] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 14:39:59 (1740685199) [ 1445.778837] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node [ 1446.338155] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 14:40:01 (1740685201) [ 1446.889273] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node [ 1447.445718] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 14:40:02 (1740685202) [ 1463.456960] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 192.168.201.104@tcp) was lost; in progress operations using this service will fail [ 1463.486882] Lustre: Evicted from MGS (at 192.168.201.104@tcp) after server handle changed from 0x3b0b1cf6b228225a to 0x3b0b1cf6b2285eac [ 1555.967802] LustreError: lustre-MDT0000-mdc-ffff8800a8d1b800: operation mds_reint to node 192.168.201.104@tcp failed: rc = -19 [ 1555.970422] LustreError: Skipped 3 previous similar messages [ 1573.632399] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 192.168.201.104@tcp) was lost; in progress operations using this service will fail [ 1573.641597] Lustre: Evicted from MGS (at 192.168.201.104@tcp) after server handle changed from 0x3b0b1cf6b2285eac to 0x3b0b1cf6b233f762 [ 1576.041101] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 14:42:10 (1740685330) [ 1576.176632] Lustre: *** cfs_fail_loc=305, val=0*** [ 1576.178856] Lustre: Skipped 4 previous similar messages [ 1608.689342] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 192.168.201.104@tcp) was lost; in progress operations using this service will fail [ 1608.699234] Lustre: Evicted from MGS (at 192.168.201.104@tcp) after server handle changed from 0x3b0b1cf6b233f762 to 0x3b0b1cf6b233fc4e [ 1613.631358] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1614.171943] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1617.517708] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 14:42:52 (1740685372) [ 1623.713594] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 192.168.201.104@tcp) was lost; in progress operations using this service will fail [ 1623.722695] Lustre: Evicted from MGS (at 192.168.201.104@tcp) after server handle changed from 0x3b0b1cf6b233fc4e to 0x3b0b1cf6b2340110 [ 1641.742321] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 14:43:16 (1740685396) [ 1648.754261] LustreError: lustre-OST0000-osc-ffff8800a8d1b800: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1660.223484] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 14:43:35 (1740685415) [ 1670.894580] LustreError: lustre-MDT0000-mdc-ffff8800a8d1b800: operation ldlm_enqueue to node 192.168.201.104@tcp failed: rc = -19 [ 1683.813643] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 192.168.201.104@tcp) was lost; in progress operations using this service will fail [ 1683.825006] Lustre: Evicted from MGS (at 192.168.201.104@tcp) after server handle changed from 0x3b0b1cf6b2340110 to 0x3b0b1cf6b2355706 [ 1689.681310] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1690.100952] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1752.023187] LustreError: lustre-MDT0000-mdc-ffff8800a8d1b800: operation mds_reint to node 192.168.201.104@tcp failed: rc = -19 [ 1765.937232] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 192.168.201.104@tcp) was lost; in progress operations using this service will fail [ 1765.995202] Lustre: Evicted from MGS (at 192.168.201.104@tcp) after server handle changed from 0x3b0b1cf6b2355706 to 0x3b0b1cf6b23d9d0b [ 1770.866242] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1771.295493] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1846.064636] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 192.168.201.104@tcp) was lost; in progress operations using this service will fail [ 1846.072376] Lustre: Evicted from MGS (at 192.168.201.104@tcp) after server handle changed from 0x3b0b1cf6b23d9d0b to 0x3b0b1cf6b245f18e [ 1852.014977] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1852.378260] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1875.763154] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 14:47:10 (1740685630) [ 1892.799196] Lustre: DEBUG MARKER: test_51: failover in 1 sec [ 1909.803321] Lustre: DEBUG MARKER: test_51: failover in 5 sec [ 1930.752858] Lustre: DEBUG MARKER: test_51: failover in 10 sec [ 1933.201771] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 192.168.201.104@tcp) was lost; in progress operations using this service will fail [ 1933.210472] LustreError: Skipped 2 previous similar messages [ 1933.217077] Lustre: Evicted from MGS (at 192.168.201.104@tcp) after server handle changed from 0x3b0b1cf6b2493d07 to 0x3b0b1cf6b2493ff4 [ 1933.222506] Lustre: Skipped 2 previous similar messages [ 1933.225718] Lustre: MGC192.168.201.104@tcp: Connection restored to 192.168.201.104@tcp (at 192.168.201.104@tcp) [ 1933.230215] Lustre: Skipped 24 previous similar messages [ 1956.675669] Lustre: DEBUG MARKER: test_51: failover in 20 sec [ 1992.637271] Lustre: DEBUG MARKER: test_51: failover in 25 sec [ 2028.297560] Lustre: 2007:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1740685648/real 1740685648] req@ffff88012c167100 x1825239268196736/t0(0) o400->lustre-MDT0000-mdc-ffff8800a8d1b800@192.168.201.104@tcp:12/10 lens 224/224 e 0 to 1 dl 1740685783 ref 1 fl Rpc:XQr/2c0/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 [ 2028.318450] Lustre: 2007:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 5 previous similar messages [ 2033.592120] Lustre: DEBUG MARKER: test_51: failover in 30 sec [ 2064.146795] LustreError: lustre-MDT0000-mdc-ffff8800a8d1b800: operation mds_reint to node 192.168.201.104@tcp failed: rc = -19 [ 2064.149552] LustreError: Skipped 3 previous similar messages [ 2064.150819] Lustre: lustre-MDT0000-mdc-ffff8800a8d1b800: Connection to lustre-MDT0000 (at 192.168.201.104@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2064.154801] Lustre: Skipped 16 previous similar messages [ 2078.417779] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 192.168.201.104@tcp) was lost; in progress operations using this service will fail [ 2078.424063] LustreError: Skipped 3 previous similar messages [ 2078.448774] Lustre: Evicted from MGS (at 192.168.201.104@tcp) after server handle changed from 0x3b0b1cf6b249466f to 0x3b0b1cf6b24cdc2c [ 2078.454147] Lustre: Skipped 3 previous similar messages [ 2101.131660] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 14:50:55 (1740685855) [ 2128.838885] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2129.257308] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 2442.464827] LustreError: lustre-OST0000-osc-ffff8800a8d1b800: operation ldlm_enqueue to node 192.168.201.104@tcp failed: rc = -19 [ 2442.467507] LustreError: Skipped 1 previous similar message [ 2459.728604] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2460.127174] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 2773.612466] Lustre: lustre-OST0000-osc-ffff8800a8d1b800: Connection to lustre-OST0000 (at 192.168.201.104@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2773.617009] Lustre: Skipped 2 previous similar messages [ 2788.466329] Lustre: lustre-OST0000-osc-ffff8800a8d1b800: Connection restored to 192.168.201.104@tcp (at 192.168.201.104@tcp) [ 2788.471870] Lustre: Skipped 7 previous similar messages [ 2790.758709] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2791.180168] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3066.200184] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 15:07:00 (1740686820) [ 3082.697542] Lustre: 23867:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1740686821/real 1740686821] req@ffff8800a7497800 x1825239369251712/t0(0) o101->lustre-MDT0000-mdc-ffff8800a8d1b800@192.168.201.104@tcp:12/10 lens 576/1024 e 0 to 1 dl 1740686837 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 [ 3085.140671] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 15:07:19 (1740686839) [ 3104.064869] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 15:07:38 (1740686858) [ 3122.991311] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 15:07:57 (1740686877) [ 3123.130233] Lustre: Mounted lustre-client [ 3148.161187] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 192.168.201.104@tcp) was lost; in progress operations using this service will fail [ 3148.170967] Lustre: Evicted from MGS (at 192.168.201.104@tcp) after server handle changed from 0x3b0b1cf6b24cdc2c to 0x3b0b1cf6b2c41e43 [ 3153.002311] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3153.547841] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3154.927483] Lustre: Unmounted lustre-client [ 3157.000967] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 15:08:31 (1740686911) [ 3247.444302] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 15:10:02 (1740687002) [ 3290.195719] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 15:10:44 (1740687044) [ 3291.455462] Lustre: Unmounted lustre-client [ 3305.978398] Lustre: Mounted lustre-client [ 3308.106161] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 15:11:02 (1740687062) [ 3308.197256] LustreError: 30294:0:(mdc_locks.c:1332:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms [ 3309.201471] LustreError: 30294:0:(mdc_locks.c:1332:mdc_finish_intent_lock()) cfs_fail_timeout interrupted [ 3309.265098] Lustre: *** cfs_fail_loc=305, val=0*** [ 3327.437733] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 15:11:22 (1740687082) [ 3327.579234] Lustre: Mounted lustre-client [ 3327.842488] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request [ 3337.888517] Lustre: Unmounted lustre-client [ 3340.017986] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 15:11:34 (1740687094) [ 3372.700873] LustreError: lustre-MDT0000-mdc-ffff8801369ac000: operation mds_reint to node 192.168.201.104@tcp failed: rc = -19 [ 3372.706047] LustreError: Skipped 1 previous similar message [ 3387.664670] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 192.168.201.104@tcp) was lost; in progress operations using this service will fail [ 3387.673438] Lustre: Evicted from MGS (at 192.168.201.104@tcp) after server handle changed from 0x3b0b1cf6b2c427f1 to 0x3b0b1cf6b2c70c9a [ 3390.699133] Lustre: lustre-MDT0000-mdc-ffff8801369ac000: Connection restored to (at 192.168.201.104@tcp) [ 3390.703213] Lustre: Skipped 13 previous similar messages [ 3428.063199] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 15:13:02 (1740687182) [ 3431.124973] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 3432.737404] Lustre: lustre-MDT0000-mdc-ffff8801369ac000: Connection to lustre-MDT0000 (at 192.168.201.104@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3432.744906] Lustre: Skipped 13 previous similar messages [ 3437.748856] LustreError: lustre-MDT0000-mdc-ffff8801369ac000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3449.602772] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 15:13:24 (1740687204) [ 3449.746902] Lustre: Mounted lustre-client [ 3454.756723] LustreError: lustre-OST0000-osc-ffff8801369ac000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3466.457545] Lustre: Unmounted lustre-client [ 3468.599440] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 15:13:43 (1740687223) [ 3473.468318] LustreError: lustre-MDT0000-mdc-ffff8801369ac000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3473.475544] LustreError: 2945:0:(file.c:6007:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5 [ 3475.980658] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 15:13:50 (1740687230) [ 3476.032777] LustreError: 3714:0:(recover.c:322:ptlrpc_recover_import()) cfs_race id 531 sleeping [ 3479.377605] LustreError: lustre-MDT0000-mdc-ffff8801369ac000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3479.384113] LustreError: 3734:0:(import.c:287:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking [ 3479.388566] LustreError: 3714:0:(recover.c:322:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=1648 [ 3479.393002] LustreError: 3714:0:(import.c:709:ptlrpc_connect_import_locked()) already connecting [ 3479.403711] LustreError: 3739:0:(file.c:6007:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108 [ 3480.437495] LustreError: 3746:0:(file.c:6007:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108 [ 3480.443024] LustreError: 3746:0:(file.c:6007:ll_inode_revalidate_fini()) Skipped 5 previous similar messages [ 3482.538338] LustreError: 3769:0:(file.c:6007:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108 [ 3482.543725] LustreError: 3769:0:(file.c:6007:ll_inode_revalidate_fini()) Skipped 13 previous similar messages [ 3486.725628] LustreError: 3817:0:(file.c:6007:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108 [ 3486.731036] LustreError: 3817:0:(file.c:6007:ll_inode_revalidate_fini()) Skipped 27 previous similar messages [ 3492.120843] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 15:14:06 (1740687246) [ 3512.687976] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3513.232745] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3517.243772] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 15:14:32 (1740687272) [ 3536.154121] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3536.714979] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3540.846325] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 15:14:55 (1740687295) [ 3585.507182] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3586.056177] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3589.441804] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 15:15:44 (1740687344) [ 3608.240400] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3608.793191] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3611.003500] Lustre: Unmounted lustre-client [ 3634.410610] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3635.897346] Lustre: Mounted lustre-client [ 3638.055413] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 15:16:32 (1740687392) [ 3638.903915] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds [ 3639.478306] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 15:16:34 (1740687394) [ 3650.148687] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 15:16:44 (1740687404) [ 3650.636631] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients [ 3651.210570] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 15:16:46 (1740687406) [ 3651.667674] Lustre: *** cfs_fail_loc=805, val=0*** [ 3654.719995] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 3671.712817] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 192.168.201.104@tcp) was lost; in progress operations using this service will fail [ 3671.719338] LustreError: Skipped 1 previous similar message [ 3671.720470] LustreError: 2007:0:(ldlm_resource.c:1125:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8800b6f62800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff88012c480d00) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 3671.734349] Lustre: Evicted from MGS (at 192.168.201.104@tcp) after server handle changed from 0x3b0b1cf6b2ca9912 to 0x3b0b1cf6b2ca9d41 [ 3671.741087] Lustre: Skipped 1 previous similar message [ 3674.889965] Lustre: Unmounted lustre-client [ 3677.395135] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 15:17:12 (1740687432) [ 3691.752821] LustreError: 2007:0:(client.c:3268:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8800aade0380 x1825239373661824/t73014444035(73014444035) o101->lustre-MDT0000-mdc-ffff8800b6f64800@192.168.201.104@tcp:12/10 lens 536/664 e 0 to 0 dl 1740687506 ref 2 fl Interpret:RPQU/204/0 rc 301/301 job:'multiop.0' uid:0 gid:0 [ 3697.767200] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3698.310028] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3701.716158] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 15:17:36 (1740687456) [ 3706.773083] LustreError: 2007:0:(osc_request.c:1027:osc_init_grant()) lustre-OST0000-osc-ffff8800b6f64800: granted 8437760 but already consumed 21020672 [ 3706.779259] LustreError: lustre-OST0000-osc-ffff8800b6f64800: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3706.786407] Lustre: 2008:0:(llite_lib.c:4135:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.104@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5) [ 3706.846256] LustreError: 15721:0:(ldlm_resource.c:1125:ldlm_resource_complain()) lustre-OST0000-osc-ffff8800b6f64800: namespace resource [0x280000401:0x4a02:0x0].0x0 (ffff8800b5d0b800) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 3706.855563] LustreError: 15721:0:(ldlm_resource.c:1125:ldlm_resource_complain()) Skipped 1 previous similar message [ 3710.224261] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 15:17:45 (1740687465) [ 3771.210545] Lustre: 16453:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1740687466/real 1740687466] req@ffff8800a691e680 x1825239373675136/t0(0) o101->lustre-MDT0000-mdc-ffff8800b6f64800@192.168.201.104@tcp:12/10 lens 576/1024 e 0 to 1 dl 1740687526 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 [ 3771.224933] Lustre: 16453:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 51 previous similar messages [ 3773.793464] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 15:18:48 (1740687528) [ 3836.683664] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 15:19:51 (1740687591) [ 3855.596962] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 15:20:10 (1740687610) [ 3918.508779] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 15:21:13 (1740687673) [ 3981.383776] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 15:22:16 (1740687736) [ 4000.312394] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 15:22:35 (1740687755) [ 4060.727612] Lustre: lustre-MDT0001-mdc-ffff8800b6f64800: Connection to lustre-MDT0001 (at 192.168.201.104@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 4060.735523] Lustre: Skipped 19 previous similar messages [ 4060.741339] Lustre: lustre-MDT0001-mdc-ffff8800b6f64800: Connection restored to (at 192.168.201.104@tcp) [ 4060.745708] Lustre: Skipped 21 previous similar messages [ 4063.048825] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 15:23:37 (1740687817) [ 4082.036991] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 15:23:56 (1740687836) [ 4100.991265] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 15:24:15 (1740687855) [ 4119.900409] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 15:24:34 (1740687874) [ 4120.011229] Lustre: Unmounted lustre-client [ 4133.974438] Lustre: Mounted lustre-client [ 4133.976423] Lustre: Skipped 1 previous similar message [ 4143.508469] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 15:24:58 (1740687898) [ 4154.363713] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 15:25:09 (1740687909) [ 4164.017570] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 192.168.201.104@tcp) was lost; in progress operations using this service will fail [ 4164.023898] LustreError: Skipped 1 previous similar message [ 4164.029428] Lustre: Evicted from MGS (at 192.168.201.104@tcp) after server handle changed from 0x3b0b1cf6b2cac7b1 to 0x3b0b1cf6b2caccf1 [ 4164.035014] Lustre: Skipped 1 previous similar message [ 4165.169900] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 15:25:19 (1740687919) [ 4188.917163] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 15:25:43 (1740687943) [ 4189.037624] Lustre: *** cfs_fail_loc=51b, val=3*** [ 4195.117027] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 15:25:49 (1740687949) [ 4195.228094] Lustre: *** cfs_fail_loc=51b, val=4*** [ 4201.326518] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 15:25:56 (1740687956) [ 4201.425035] Lustre: *** cfs_fail_loc=50f, val=3*** [ 4204.946501] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 15:25:59 (1740687959) [ 4205.038326] Lustre: *** cfs_fail_loc=50f, val=4*** [ 4208.562987] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 15:26:03 (1740687963) [ 4208.664971] Lustre: *** cfs_fail_loc=510, val=3*** [ 4212.178686] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 15:26:06 (1740687966) [ 4212.291728] Lustre: *** cfs_fail_loc=51b, val=3*** [ 4218.349203] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 15:26:13 (1740687973) [ 4279.558672] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 15:27:14 (1740688034) [ 4279.657948] LustreError: 32083:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms [ 4281.964118] LustreError: lustre-MDT0000-mdc-ffff8800a8e50800: operation ldlm_enqueue to node 192.168.201.104@tcp failed: rc = -107 [ 4281.969419] LustreError: Skipped 2 previous similar messages [ 4281.975169] LustreError: lustre-MDT0000-mdc-ffff8800a8e50800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 4281.982909] LustreError: 32099:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a8e50800: inode [0x20000afe2:0x3:0x0] mdc close failed: rc = -108 [ 4281.993923] LustreError: 32099:0:(ldlm_resource.c:1125:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8800a8e50800: namespace resource [0x20000afe2:0xc:0x0].0xc (ffff8800b6e31300) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 4281.997988] LustreError: 32104:0:(file.c:6007:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108 [ 4281.997991] LustreError: 32104:0:(file.c:6007:ll_inode_revalidate_fini()) Skipped 20 previous similar messages [ 4283.662512] LustreError: 32083:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake [ 4285.693071] LustreError: 32128:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms [ 4288.014761] LustreError: lustre-MDT0000-mdc-ffff8800a8e50800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 4288.023935] LustreError: 32144:0:(ldlm_resource.c:1125:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8800a8e50800: namespace resource [0x20000afe2:0xc:0x0].0xc (ffff8800b6cb1400) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 4288.033493] LustreError: 32144:0:(ldlm_resource.c:1125:ldlm_resource_complain()) Skipped 1 previous similar message [ 4288.038398] LustreError: 32149:0:(file.c:6007:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108 [ 4288.044307] LustreError: 32149:0:(file.c:6007:ll_inode_revalidate_fini()) Skipped 3 previous similar messages [ 4289.697504] LustreError: 32128:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake [ 4296.051046] LustreError: lustre-MDT0000-mdc-ffff8800a8e50800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 4296.141696] LustreError: 32190:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms [ 4298.461304] LustreError: lustre-MDT0000-mdc-ffff8800a8e50800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 4298.485305] LustreError: 32211:0:(file.c:6007:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108 [ 4300.143508] LustreError: 32190:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake [ 4300.148576] LustreError: 32190:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a8e50800: inode [0x20000afe2:0xc:0x0] mdc close failed: rc = -108 [ 4300.155196] LustreError: 32190:0:(file.c:247:ll_close_inode_openhandle()) Skipped 4 previous similar messages [ 4304.798919] LustreError: 32270:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms [ 4307.125480] LustreError: lustre-MDT0000-mdc-ffff8800a8e50800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 4307.133826] LustreError: 32286:0:(ldlm_resource.c:1125:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8800a8e50800: namespace resource [0x20000afe2:0xc:0x0].0xc (ffff8800b6df9800) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 4307.143455] LustreError: 32286:0:(ldlm_resource.c:1125:ldlm_resource_complain()) Skipped 1 previous similar message [ 4307.149945] LustreError: 32291:0:(file.c:6007:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108 [ 4307.155742] LustreError: 32291:0:(file.c:6007:ll_inode_revalidate_fini()) Skipped 26 previous similar messages [ 4308.803492] LustreError: 32270:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake [ 4315.157578] LustreError: lustre-MDT0000-mdc-ffff8800a8e50800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 4315.294391] LustreError: 32333:0:(ldlm_flock.c:858:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms [ 4317.607532] LustreError: lustre-MDT0000-mdc-ffff8800a8e50800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 4317.615950] LustreError: 32349:0:(ldlm_resource.c:1125:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8800a8e50800: namespace resource [0x20000afe2:0xc:0x0].0xc (ffff8800ac15b700) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 4317.625394] LustreError: 32349:0:(ldlm_resource.c:1125:ldlm_resource_complain()) Skipped 1 previous similar message [ 4317.631745] LustreError: 32354:0:(file.c:6007:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108 [ 4319.298517] LustreError: 32333:0:(ldlm_flock.c:858:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake [ 4321.644555] LustreError: lustre-MDT0000-mdc-ffff8800a8e50800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 4323.434036] LustreError: 32363:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a8e50800: inode [0x20000afe2:0xc:0x0] mdc close failed: rc = -108 [ 4329.979821] LustreError: 32438:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms [ 4329.984893] LustreError: 32438:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message [ 4332.302038] LustreError: lustre-MDT0000-mdc-ffff8800a8e50800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 4333.989499] LustreError: 32438:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake [ 4333.993934] LustreError: 32438:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message [ 4333.999084] LustreError: 32438:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a8e50800: inode [0x20000afe2:0xc:0x0] mdc close failed: rc = -108 [ 4334.399894] LustreError: 32483:0:(file.c:6007:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108 [ 4334.405304] LustreError: 32483:0:(file.c:6007:ll_inode_revalidate_fini()) Skipped 40 previous similar messages [ 4338.742067] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 15:28:13 (1740688093) [ 4399.238569] Lustre: 762:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1740688094/real 1740688094] req@ffff8800a7495c00 x1825239373886336/t0(0) o101->lustre-MDT0000-mdc-ffff8800a8e50800@192.168.201.104@tcp:12/10 lens 576/1024 e 0 to 1 dl 1740688154 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'stat.0' uid:0 gid:0 [ 4399.252681] Lustre: 762:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 6 previous similar messages [ 4399.667424] LustreError: 27042:0:(ldlm_lockd.c:2943:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms [ 4399.672411] LustreError: 27042:0:(ldlm_lockd.c:2943:ldlm_bl_thread_blwi()) Skipped 1 previous similar message [ 4403.676503] LustreError: 27042:0:(ldlm_lockd.c:2943:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake [ 4403.680814] LustreError: 27042:0:(ldlm_lockd.c:2943:ldlm_bl_thread_blwi()) Skipped 1 previous similar message [ 4405.954479] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 15:29:20 (1740688160) [ 4468.769741] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 15:30:23 (1740688223) [ 4531.553501] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 15:31:26 (1740688286) [ 4557.056330] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 15:31:51 (1740688311) [ 4557.200675] Lustre: Mounted lustre-client [ 4677.988496] Lustre: Unmounted lustre-client [ 4680.133396] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 15:33:54 (1740688434) [ 4680.331175] LustreError: 5003:0:(osc_request.c:2941:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 4000ms [ 4682.150754] Lustre: lustre-OST0000-osc-ffff8800a8e50800: Connection to lustre-OST0000 (at 192.168.201.104@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 4682.158609] Lustre: Skipped 14 previous similar messages [ 4682.164180] LustreError: lustre-OST0000-osc-ffff8800a8e50800: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4682.171661] LustreError: 5091:0:(ldlm_resource.c:1125:ldlm_resource_complain()) lustre-OST0000-osc-ffff8800a8e50800: namespace resource [0x280000401:0x4a2d:0x0].0x0 (ffff8800b6f1b400) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 4682.182266] LustreError: 5091:0:(ldlm_resource.c:1125:ldlm_resource_complain()) Skipped 1 previous similar message [ 4684.335516] LustreError: 5003:0:(osc_request.c:2941:osc_build_rpc()) cfs_fail_timeout id 414 awake [ 4684.339917] Lustre: 2011:0:(llite_lib.c:4135:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.104@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108) [ 4684.351863] Lustre: lustre-OST0000-osc-ffff8800a8e50800: Connection restored to (at 192.168.201.104@tcp) [ 4684.356535] Lustre: Skipped 14 previous similar messages [ 4686.465223] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 15:34:01 (1740688441) [ 4750.094289] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 15:35:04 (1740688504) [ 4750.592814] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1 [ 4751.177447] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 15:35:05 (1740688505) [ 4775.861533] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136 [ 4776.426735] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 15:35:31 (1740688531) [ 4799.457269] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 15:35:54 (1740688554) [ 4896.562142] LustreError: lustre-MDT0000-mdc-ffff8801369a9800: operation mds_connect to node 192.168.201.104@tcp failed: rc = -16 [ 4896.567511] LustreError: Skipped 9 previous similar messages [ 4901.569940] Lustre: Mounted lustre-client [ 4903.778770] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 15:37:38 (1740688658) [ 4911.585441] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 192.168.201.104@tcp) was lost; in progress operations using this service will fail [ 4911.595883] Lustre: Evicted from MGS (at 192.168.201.104@tcp) after server handle changed from 0x3b0b1cf6b2cae7e0 to 0x3b0b1cf6b2caebc2 [ 4911.802715] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 15:37:46 (1740688666) [ 4921.533351] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 15:37:56 (1740688676) [ 4926.262443] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 4948.625045] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4949.167875] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4953.302335] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 15:38:28 (1740688708) [ 4954.123493] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree [ 4954.716578] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 15:38:29 (1740688709) [ 4963.116669] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 15:38:37 (1740688717) [ 4989.387264] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 15:39:04 (1740688744) [ 5008.638018] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5009.113083] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5090.606207] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5091.157180] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5112.413689] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5112.970050] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5128.451128] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 15:41:23 (1740688883) [ 5148.716580] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5149.282293] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5170.377068] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5170.787223] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5172.389547] Lustre: DEBUG MARKER: recovery-small test_144b: @@@@@@ FAIL: Create thread still running