[ 5198.658382] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5206.924505] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 5214.970334] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5216.435562] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5217.974303] Lustre: Failing over lustre-MDT0000
[ 5218.273140] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5218.277735] LustreError: 77831:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5218.283082] Lustre: Skipped 4 previous similar messages
[ 5218.310034] LustreError: 77831:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages
[ 5218.353408] Lustre: server umount lustre-MDT0000 complete
[ 5234.656126] Lustre: 3643:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788454476/real 1788454476]  req@ffff8b9182830000 x1875325075403648/t0(0) o400->MGC192.168.206.127@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788454492 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5234.668145] Lustre: 3643:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5234.680986] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5243.270477] LDISKFS-fs (dm-0): 7 truncates cleaned up
[ 5243.275090] LDISKFS-fs (dm-0): recovery complete
[ 5243.301873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5244.576632] LustreError: 146010:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0)
[ 5244.588807] LustreError: 146010:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list  req@ffff8b9182830a80 x1875325075409024/t0(0) o250->MGC192.168.206.127@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1788454502 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5244.628551] LustreError: 146010:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out.
[ 5245.986511] LustreError: 3640:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight  req@ffff8b92b61c4380 x1875325075412736/t0(0) o250->MGC192.168.206.127@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5246.464673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5246.513152] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5247.695144] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 5251.577475] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 5251.672653] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted.
[ 5251.725280] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:619 to 0x2c0000401:641)
[ 5251.727225] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:685 to 0x280000401:705)
[ 5251.852663] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5261.631252] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5263.567535] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5268.995581] Lustre: DEBUG MARKER: (dd_pid=131127, time=0, timeout=600)
[ 5294.388418] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5300.782989] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 5310.150081] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5312.182879] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5314.810966] Lustre: Failing over lustre-MDT0000
[ 5315.707041] Lustre: server umount lustre-MDT0000 complete
[ 5318.117503] LustreError: 7786:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5318.151205] LustreError: 7786:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 37 previous similar messages
[ 5335.008316] Lustre: 3644:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788454576/real 1788454576]  req@ffff8b92bb44ca80 x1875325075469312/t0(0) o400->MGC192.168.206.127@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788454592 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5335.050255] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5338.457573] LDISKFS-fs (dm-0): 3 truncates cleaned up
[ 5338.459343] LDISKFS-fs (dm-0): recovery complete
[ 5338.479302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5345.709084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5345.786541] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5346.355795] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 5350.892904] LustreError: 3640:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-lwp-OST0001: namespace resource [0x200000006:0x20000:0x0].0x0 (ffff8b91827ef300) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5350.931470] LustreError: 3640:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 5 previous similar messages
[ 5351.013150] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted.
[ 5351.073703] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:707 to 0x280000401:737)
[ 5351.073943] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:619 to 0x2c0000401:673)
[ 5351.959255] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5363.247857] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5365.245096] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5371.109614] Lustre: DEBUG MARKER: (dd_pid=133513, time=0, timeout=600)
[ 5415.393570] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 12:57:52 (1788454672)
[ 5464.579869] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 12:58:41 (1788454721)
[ 5492.979519] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 5509.365400] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 5511.457374] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 5513.540641] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 5515.795675] Lustre: DEBUG MARKER: Set quota for 1 times
[ 5519.649634] Lustre: DEBUG MARKER: Set quota for 2 times
[ 5523.000165] Lustre: DEBUG MARKER: Set quota for 3 times
[ 5527.257695] Lustre: DEBUG MARKER: Set quota for 4 times
[ 5531.169439] Lustre: DEBUG MARKER: Set quota for 5 times
[ 5534.674616] Lustre: DEBUG MARKER: Set quota for 6 times
[ 5537.942872] Lustre: DEBUG MARKER: Set quota for 7 times
[ 5541.307077] Lustre: DEBUG MARKER: Set quota for 8 times
[ 5544.442898] Lustre: DEBUG MARKER: Set quota for 9 times
[ 5572.106801] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 13:00:28 (1788454828)
[ 5586.928903] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 5586.942560] LustreError: Skipped 3 previous similar messages
[ 5586.956114] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5586.972311] Lustre: Skipped 4 previous similar messages
[ 5586.977420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5591.529595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5591.544140] Lustre: Skipped 3 previous similar messages
[ 5595.194104] Lustre: server umount lustre-MDT0000 complete
[ 5596.644842] LustreError: 6520:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5596.666304] LustreError: 6520:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 33 previous similar messages
[ 5598.750979] LustreError: 6501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454856 with bad export cookie 12599905553923736811
[ 5598.761875] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5598.765798] LustreError: 6501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 5599.138894] Lustre: server umount lustre-MDT0001 complete
[ 5614.398472] Lustre: server umount lustre-OST0000 complete
[ 5617.122870] Lustre: 3643:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788454859/real 1788454859]  req@ffff8b92b2250700 x1875325075670912/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788454875 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5618.297401] Lustre: server umount lustre-OST0001 complete
[ 5637.728567] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local
[ 5648.538528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5649.257713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5653.254687] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5661.503987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5661.845619] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 5667.186259] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5670.886507] Lustre: 159115:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 5677.956719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 5678.427330] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 5684.445523] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5686.636735] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97)
[ 5693.378523] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 5693.603836] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 5694.645046] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:740 to 0x280000401:769)
[ 5694.658729] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:675 to 0x2c0000401:705)
[ 5699.843665] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5709.806719] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 5714.649092] Lustre: 160995:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 5735.918145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5735.936767] Lustre: Skipped 3 previous similar messages
[ 5741.044754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5741.216347] Lustre: server umount lustre-MDT0000 complete
[ 5745.145764] LustreError: 157942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788455003 with bad export cookie 12599905553923747794
[ 5745.148792] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5745.153061] LustreError: 157942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message
[ 5745.454518] Lustre: server umount lustre-MDT0001 complete
[ 5760.164189] Lustre: server umount lustre-OST0000 complete
[ 5763.040775] Lustre: 3641:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788455004/real 1788455004]  req@ffff8b92b13e7480 x1875325075752960/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788455020 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5764.729229] Lustre: server umount lustre-OST0001 complete
[ 5782.381829] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local
[ 5793.769440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5794.237639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5798.953944] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5808.053179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5812.835129] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5815.958957] Lustre: 164737:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 5823.167219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 5830.081735] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5833.715874] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:129)
[ 5839.189948] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 5839.386121] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 5839.391386] Lustre: Skipped 2 previous similar messages
[ 5840.440744] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:740 to 0x280000401:801)
[ 5840.453894] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:675 to 0x2c0000401:737)
[ 5845.998479] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5854.927493] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 5858.825797] Lustre: 166609:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 5871.269609] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 13:05:28 (1788455128)
[ 5873.351378] Lustre: DEBUG MARKER: OST0_SIZE: 3605396 required: 6144
[ 5875.037921] Lustre: DEBUG MARKER: run for 4MB test file
[ 5887.462575] Lustre: DEBUG MARKER: User quota (limit: 4 MB)
[ 5894.702372] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT
[ 5896.648094] Lustre: DEBUG MARKER: Write half of file
[ 5898.845108] Lustre: DEBUG MARKER: Write out of block quota ...
[ 5901.403758] Lustre: DEBUG MARKER: Step1: done
[ 5903.805589] Lustre: DEBUG MARKER: Step2: rewrite should succeed
[ 5906.217604] Lustre: DEBUG MARKER: Step2: done
[ 5935.285959] Lustre: DEBUG MARKER: OST0_SIZE: 3605396 required: 61440
[ 5937.182614] Lustre: DEBUG MARKER: run for 40MB test file
[ 5950.673926] Lustre: DEBUG MARKER: User quota (limit: 40 MB)
[ 5957.487727] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT
[ 5959.253236] Lustre: DEBUG MARKER: Write half of file
[ 5963.102973] Lustre: DEBUG MARKER: Write out of block quota ...
[ 5966.276243] Lustre: DEBUG MARKER: Step1: done
[ 5968.055295] Lustre: DEBUG MARKER: Step2: rewrite should succeed
[ 5969.779562] Lustre: DEBUG MARKER: Step2: done
[ 6026.113690] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 13:08:02 (1788455282)
[ 6074.413491] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 13:08:50 (1788455330)
[ 6119.140525] Lustre: DEBUG MARKER: Write...
[ 6121.593514] Lustre: DEBUG MARKER: Write out of block quota ...
[ 6174.286861] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 13:10:31 (1788455431)
[ 6181.858927] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 13:10:38 (1788455438)
[ 6192.778938] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 13:10:49 (1788455449)
[ 6202.556749] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 13:10:59 (1788455459)
[ 6212.362473] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 13:11:08 (1788455468)
[ 6266.753858] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 6398.407569] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 6549.733033] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 13:16:46 (1788455806)
[ 6606.388547] Lustre: DEBUG MARKER: Restart...
[ 6612.449503] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 6612.453954] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6612.463557] Lustre: Skipped 9 previous similar messages
[ 6612.468348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6612.477560] Lustre: Skipped 3 previous similar messages
[ 6616.545927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6618.201516] Lustre: server umount lustre-MDT0000 complete
[ 6621.690869] LustreError: 163594:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6621.718759] LustreError: 163594:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages
[ 6622.564555] LustreError: 167345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788455880 with bad export cookie 12599905553923750097
[ 6622.566325] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6622.591205] LustreError: 167345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages
[ 6622.882439] Lustre: server umount lustre-MDT0001 complete
[ 6632.416345] LustreError: 3643:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED  req@ffff8b9189390000 x1875325076245888/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 6632.426083] LustreError: 3643:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8  qsd:lustre-OST0000 qtype:usr lqe: ffff8b9185060480 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 1596 qunit:0 qtune:0 edquot:0 default:no revoke:0
[ 6636.540716] Lustre: server umount lustre-OST0000 complete
[ 6651.078298] Lustre: server umount lustre-OST0001 complete
[ 6669.747568] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local
[ 6680.864855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 6681.339543] LustreError: 189775:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6681.484238] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6686.549208] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6695.089683] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 6695.462053] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 6700.059238] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6703.228232] Lustre: 190919:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6710.447863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 6717.359454] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6720.110781] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:161)
[ 6725.228935] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 6725.425373] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6725.431281] Lustre: Skipped 1 previous similar message
[ 6726.448550] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:813 to 0x280000401:833)
[ 6726.506341] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:745 to 0x2c0000401:769)
[ 6731.077653] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6738.683648] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6743.015322] Lustre: 192792:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6807.686513] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 13:21:04 (1788456064)
[ 6866.187672] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 13:22:02 (1788456122)
[ 8651.202806] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 13:51:47 (1788457907)
[ 8664.038352] LustreError: lustre-MDT0000-lwp-MDT0001: operation quota_acquire to node 0@lo failed: rc = -107
[ 8664.051521] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 8664.089694] Lustre: Skipped 3 previous similar messages
[ 8664.105545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 8664.108166] Lustre: Skipped 3 previous similar messages
[ 8666.593175] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 8666.612913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 8669.883216] Lustre: server umount lustre-MDT0000 complete
[ 8672.232952] LustreError: 189772:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 8672.267934] LustreError: 189772:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages
[ 8674.314917] LustreError: 189756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788457932 with bad export cookie 12599905553923759232
[ 8674.316503] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 8674.364708] LustreError: 189756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages
[ 8674.728724] Lustre: server umount lustre-MDT0001 complete
[ 8689.969475] Lustre: server umount lustre-OST0000 complete
[ 8693.472198] Lustre: 3644:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788457935/real 1788457935]  req@ffff8b92bf448700 x1875325079714816/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788457951 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 8693.489655] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 8693.514888] Lustre: Skipped 3 previous similar messages
[ 8694.036987] Lustre: server umount lustre-OST0001 complete
[ 8713.041303] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local
[ 8728.136864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 8728.630974] LustreError: 200423:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 8728.732887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 8734.822676] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8745.433991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 8746.084158] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 8750.496570] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8753.973583] Lustre: 201564:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 8760.944576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 8761.469069] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 8767.919735] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8770.661923] LustreError: 201920:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 8770.676640] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:193)
[ 8770.686036] LustreError: 201920:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages
[ 8773.751317] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5836 to 0x280000401:5857)
[ 8777.724666] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 8778.018096] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 8783.379606] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5770 to 0x2c0000401:5793)
[ 8787.315444] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8797.453435] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 8802.735565] Lustre: 203439:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 8832.623068] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 13:54:49 (1788458089)
[ 8864.148316] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 13:55:20 (1788458120)
[ 8891.763637] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 13:55:48 (1788458148)
[ 8925.391645] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 13:56:22 (1788458182)
[ 8962.978864] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 13:56:59 (1788458219)
[ 9047.500847] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 13:58:24 (1788458304)
[ 9082.919160] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 13:58:59 (1788458339)
[ 9153.648775] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 14:00:10 (1788458410)
[ 9160.175079] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9160.178552] Lustre: Skipped 2 previous similar messages
[ 9162.199380] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9162.204056] Lustre: Skipped 63 previous similar messages
[ 9166.200685] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9166.202814] Lustre: Skipped 145 previous similar messages
[ 9174.206237] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9174.208513] Lustre: Skipped 323 previous similar messages
[ 9190.215494] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9190.217456] Lustre: Skipped 582 previous similar messages
[ 9222.251084] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9222.255678] Lustre: Skipped 1256 previous similar messages
[ 9286.263838] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9286.265655] Lustre: Skipped 2357 previous similar messages
[ 9749.473660] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 9749.478696] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 9749.497101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 9749.506882] Lustre: Skipped 4 previous similar messages
[ 9751.012076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 9751.014886] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 9751.044203] Lustre: Skipped 2 previous similar messages
[ 9754.860535] Lustre: server umount lustre-MDT0000 complete
[ 9756.131148] LustreError: 200423:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9756.142856] LustreError: 200423:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages
[ 9758.469334] LustreError: 200403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788459016 with bad export cookie 12599905553925539815
[ 9758.476314] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 9758.480273] LustreError: 200403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 9761.249916] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 9761.252582] LustreError: 201876:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9761.256508] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping)
[ 9761.256516] Lustre: Skipped 3 previous similar messages
[ 9761.284946] Lustre: Skipped 1 previous similar message
[ 9761.300820] LustreError: 201876:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message
[ 9763.297791] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping)
[ 9763.307169] Lustre: Skipped 2 previous similar messages
[ 9764.959555] Lustre: server umount lustre-MDT0001 complete
[ 9769.844728] Lustre: server umount lustre-OST0000 complete
[ 9774.458493] Lustre: server umount lustre-OST0001 complete
[ 9782.105354] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid
[ 9791.772331] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local
[ 9842.156221] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local
[ 9851.968211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 9852.148871] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000
[ 9852.191471] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[ 9852.262566] Lustre: lustre-MDT0000: new disk, initializing
[ 9852.349127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 9852.365878] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[ 9857.112787] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9868.777679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 9868.927596] Lustre: 219822:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17
[ 9868.988240] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
[ 9868.998470] Lustre: Skipped 1 previous similar message
[ 9869.070452] Lustre: lustre-MDT0001: new disk, initializing
[ 9869.116033] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 9869.142078] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt
[ 9869.152596] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt]
[ 9873.539868] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9878.674936] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params
[ 9885.379114] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 9885.715870] Lustre: lustre-OST0000: new disk, initializing
[ 9885.723476] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[ 9885.728496] Lustre: 221455:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0]
[ 9885.796401] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 9887.330470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost
[ 9887.349494] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost]
[ 9887.447209] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401
[ 9891.738467] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9903.605978] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 9903.831996] Lustre: lustre-OST0001: new disk, initializing
[ 9903.837519] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[ 9903.854032] Lustre: 222327:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0]
[ 9903.982993] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 9905.816993] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost
[ 9905.825705] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost]
[ 9905.888562] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401
[ 9911.069459] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9921.473522] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 9925.395433] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params
[ 9953.400543] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 14:13:30 (1788459210)
[ 9959.532214] Lustre: DEBUG MARKER: == sanity-quota test 49c: lfs quota long options don't consume an extra argument ========================================================== 14:13:36 (1788459216)
[ 9966.988947] Lustre: DEBUG MARKER: == sanity-quota test 49d: lfs quota -d and --delimiter both work ========================================================== 14:13:43 (1788459223)
[ 9974.573425] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 14:13:51 (1788459231)
[10010.499672] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 14:14:27 (1788459267)
[10060.035457] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 14:15:17 (1788459317)
[10085.486392] Lustre: DEBUG MARKER: rename directory return 255
[10128.114598] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 14:16:24 (1788459384)
[10151.574494] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 14:16:48 (1788459408)
[10186.759351] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 14:17:23 (1788459443)
[10308.485988] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 14:19:24 (1788459564)
[10339.192346] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 14:19:56 (1788459596)
[10373.443538] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 14:20:30 (1788459630)
[10521.465640] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 14:22:58 (1788459778)
[10525.666971] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[10525.670217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[10525.676631] Lustre: Skipped 1 previous similar message
[10525.681866] Lustre: Skipped 1 previous similar message
[10530.793764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[10530.800837] Lustre: Skipped 4 previous similar messages
[10531.604991] Lustre: server umount lustre-MDT0000 complete
[10535.138738] LustreError: 219813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788459793 with bad export cookie 12599905553925912201
[10535.141142] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[10535.149388] LustreError: 219813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages
[10535.413715] Lustre: server umount lustre-MDT0001 complete
[10548.306386] Lustre: server umount lustre-OST0000 complete
[10551.267357] Lustre: 3644:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788459793/real 1788459793]  req@ffff8b9285c20700 x1875325086852352/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788459809 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[10551.288847] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[10551.302649] Lustre: Skipped 2 previous similar messages
[10552.316303] Lustre: server umount lustre-OST0001 complete
[10572.246148] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local
[10582.257308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[10582.652625] LustreError: 239862:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[10582.675586] LustreError: 239862:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages
[10582.766735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[10582.780482] LustreError: lustre-MDT0000: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[10587.077072] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10588.132566] LustreError: 239863:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[10592.226041] LustreError: 239862:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[10595.393200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[10595.750581] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[10595.760638] LustreError: lustre-MDT0001: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[10600.431916] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10603.525691] Lustre: 241009:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[10609.805608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[10610.118958] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[10610.129178] LustreError: lustre-OST0000: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[10617.704393] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10621.419474] LustreError: 241362:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[10627.579034] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:20 to 0x280000401:65)
[10627.974628] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[10628.360139] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[10628.389651] LustreError: lustre-OST0001: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[10633.710795] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:20 to 0x2c0000401:65)
[10635.869410] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10643.160741] Lustre: DEBUG MARKER: Using TIMEOUT=20
[10646.814815] Lustre: 242882:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[10654.444441] LustreError: 239859:0:(osd_handler.c:3425:osd_quota_transfer()) lustre-MDT0000: quota transfer failed. Is project enforcement enabled on the ldiskfs filesystem? rc = -95
[10658.785057] LustreError: lustre-MDT0000-lwp-MDT0001: operation quota_acquire to node 0@lo failed: rc = -107
[10658.790842] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[10658.791027] LustreError: Skipped 1 previous similar message
[10658.823854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[10664.420032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[10664.426300] Lustre: Skipped 4 previous similar messages
[10664.878273] Lustre: server umount lustre-MDT0000 complete
[10668.129994] LustreError: 239844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788459926 with bad export cookie 12599905553925961110
[10668.132629] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[10668.145128] LustreError: 239844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message
[10668.422507] Lustre: server umount lustre-MDT0001 complete
[10681.411219] Lustre: server umount lustre-OST0000 complete
[10691.552724] LustreError: 3642:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED  req@ffff8b9183b0e680 x1875325086908928/t0(0) o601->lustre-MDT0000-lwp-OST0001@0@lo:23/10 lens 336/336 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[10691.592418] LustreError: 3642:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8  qsd:lustre-OST0001 qtype:grp lqe: ffff8b918727c480 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 1516 qunit:0 qtune:0 edquot:0 default:no revoke:0
[10691.614384] LustreError: 3642:0:(qsd_handler.c:298:qsd_req_completion()) Skipped 1 previous similar message
[10694.520860] Lustre: server umount lustre-OST0001 complete
[10715.717313] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local
[10725.618134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[10726.141127] LustreError: 245270:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[10726.161464] LustreError: 245270:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages
[10726.215595] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[10730.807203] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10739.864760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[10745.077070] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10748.514035] Lustre: 246417:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[10756.025027] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[10762.598556] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10765.484263] LustreError: 246772:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[10765.504522] LustreError: 246772:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages
[10770.444289] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[10771.771145] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97)
[10771.780268] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:20 to 0x2c0000401:97)
[10777.006722] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10784.383205] Lustre: DEBUG MARKER: Using TIMEOUT=20
[10787.884195] Lustre: 248295:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[10813.643676] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 14:27:50 (1788460070)
[10864.010941] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[10866.182633] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 14:28:42 (1788460122)
[10888.513468] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[10890.210539] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 14:29:07 (1788460147)
[10931.877540] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[10933.412357] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 14:29:50 (1788460190)
[10967.340530] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 14:30:24 (1788460224)
[10977.938551] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[10983.295434] Lustre: 254955:0:(qsd_reint.c:249:qsd_reint_index()) lustre-OST0000: index version for fid [0x200000005:0x1007:0x0] is 0, but index isn't empty (1)
[10987.708701] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[10993.528954] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[10997.333786] Lustre: DEBUG MARKER: Write...
[11016.810383] LustreError: 256300:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[11022.453770] Lustre: DEBUG MARKER: Write...
[11033.816371] Lustre: DEBUG MARKER: Write...
[11108.005925] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 14:32:44 (1788460364)
[11131.014700] LustreError: 260319:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[11173.632545] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 14:33:50 (1788460430)
[11201.310985] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[11203.242265] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[11291.693276] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 14:35:48 (1788460548)
[11339.565450] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 14:36:36 (1788460596)
[11374.096490] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 14:37:11 (1788460631)
[11384.422473] Lustre: DEBUG MARKER: User quota (block hardlimit:100 MB)
[11406.169433] Lustre: DEBUG MARKER: Write...
[11407.991303] Lustre: DEBUG MARKER: Write out of block quota ...
[11499.822672] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 14:39:16 (1788460756)
[11512.357676] Lustre: DEBUG MARKER: User quota (block hardlimit:1000 MB)
[11538.595734] LustreError: 245269:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool2 id:60000 enforced:1 hard:10240 soft:0 granted:10240 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[11597.781710] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 14:40:54 (1788460854)
[11609.410901] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB)
[11629.945594] Lustre: DEBUG MARKER: Write...
[11632.082794] Lustre: DEBUG MARKER: Write out of block quota ...
[11684.964626] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 14:42:21 (1788460941)
[11713.432519] Lustre: DEBUG MARKER: set to use default quota
[11715.023253] Lustre: DEBUG MARKER: set default quota
[11716.308673] Lustre: DEBUG MARKER: get default quota
[11722.934907] Lustre: DEBUG MARKER: Test not out of quota
[11726.269376] Lustre: DEBUG MARKER: Test out of quota
[11737.477986] Lustre: DEBUG MARKER: Increase default quota
[11763.374265] Lustre: DEBUG MARKER: Set quota to override default quota
[11763.466850] LustreError: 245266:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45056 time:1789065821 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[11776.587854] Lustre: DEBUG MARKER: Set to use default quota again
[11795.602679] Lustre: DEBUG MARKER: Cleanup
[11857.200431] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 14:45:14 (1788461114)
[11878.362887] Lustre: DEBUG MARKER: set default quota for qpool1
[11879.983900] Lustre: DEBUG MARKER: Write from user that hasn't lqe
[11925.208677] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 14:46:22 (1788461182)
[12003.237491] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 14:47:40 (1788461260)
[12062.772666] Lustre: DEBUG MARKER: Write...
[12065.313781] Lustre: DEBUG MARKER: Write out of block quota ...
[12178.245434] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 14:50:35 (1788461435)
[12212.365233] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 14:51:08 (1788461468)
[12220.731687] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 14:51:17 (1788461477)
[12264.915376] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 14:52:01 (1788461521)
[12315.449438] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 14:52:52 (1788461572)
[12337.389350] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 14:53:14 (1788461594)
[12365.126813] Lustre: *** cfs_fail_loc=a06, val=0***
[12365.133849] Lustre: Skipped 3267 previous similar messages
[12365.528554] LustreError: 245284:0:(qmt_lock.c:479:qmt_lvbo_update()) $$$ failed to release quota space on glimpse 0!=2048 : rc = -11
                 qmt:lustre-QMT0000 pool:dt-0x0 id:60000 enforced:1 hard:102400 soft:0 granted:25600 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no
[12371.093256] Lustre: Failing over lustre-OST0001
[12371.225561] Lustre: server umount lustre-OST0001 complete
[12372.209602] LustreError: 246773:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[12374.502835] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[12374.512503] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107
[12374.516328] Lustre: Skipped 3 previous similar messages
[12377.293701] LustreError: 247662:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[12377.327938] LustreError: 247662:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages
[12380.161424] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[12380.490338] Lustre: lustre-OST0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180
[12380.514150] Lustre: lustre-OST0001: in recovery but waiting for the first client to connect
[12381.845971] Lustre: lustre-OST0001: Will be in recovery for at least 1:00, or until 3 clients reconnect
[12381.938026] Lustre: lustre-OST0001: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted.
[12381.940668] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo)
[12381.961764] Lustre: Skipped 7 previous similar messages
[12387.150934] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[12435.232434] Lustre: DEBUG MARKER: == sanity-quota test 81: Race qmt_start_pool_recalc with qmt_pool_free ========================================================== 14:54:52 (1788461692)
[12446.251398] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[12453.877725] LustreError: 299854:0:(qmt_pool.c:1406:qmt_pool_recalc()) cfs_fail_timeout id a07 sleeping for 10000ms
[12459.250120] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.27@tcp (stopping)
[12459.260565] Lustre: Skipped 3 previous similar messages
[12460.517095] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[12460.517220] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[12460.526468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[12460.541022] Lustre: Skipped 4 previous similar messages
[12463.896120] LustreError: 299854:0:(qmt_pool.c:1406:qmt_pool_recalc()) cfs_fail_timeout id a07 awake
[12464.180298] Lustre: server umount lustre-MDT0000 complete
[12464.345331] LustreError: 254876:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[12464.366671] LustreError: 254876:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages
[12473.031030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[12473.171995] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[12473.520336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[12473.524792] Lustre: Skipped 3 previous similar messages
[12473.580864] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:113 to 0x2c0000401:129)
[12473.581752] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:112 to 0x280000401:129)
[12478.489839] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[12478.947656] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo)
[12478.958371] Lustre: Skipped 1 previous similar message
[12508.319477] Lustre: DEBUG MARKER: == sanity-quota test 82: verify more than 8 qids for single operation ========================================================== 14:56:05 (1788461765)
[12533.720934] Lustre: DEBUG MARKER: == sanity-quota test 83: Setting default quota shouldn't affect grace time ========================================================== 14:56:30 (1788461790)
[12556.530835] Lustre: DEBUG MARKER: == sanity-quota test 84: Reset quota should fix the insane granted quota ========================================================== 14:56:53 (1788461813)
[12605.570656] Lustre: *** cfs_fail_loc=a08, val=0***
[12605.573179] Lustre: Skipped 2674 previous similar messages
[12605.577818] Lustre: *** cfs_fail_loc=a08, val=0***
[12605.588767] Lustre: Skipped 1 previous similar message
[12689.172182] Lustre: DEBUG MARKER: == sanity-quota test 85: do not hung at write with the least_qunit ========================================================== 14:59:06 (1788461946)
[12713.775799] LustreError: 245269:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:3072 soft:0 granted:3072 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[12765.122722] Lustre: DEBUG MARKER: == sanity-quota test 86: Pre-acquired quota should be released if quota is over limit ========================================================== 15:00:22 (1788462022)
[12829.439622] LustreError: 245265:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:16384 time:1789066887 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[12977.358913] LustreError: 245265:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:16384 time:1789067035 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[12988.159499] Lustre: DEBUG MARKER: sanity-quota test_86: @@@@@@ FAIL: succeed to create files, expect failed