[ 5036.236981] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5043.560759] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 5052.637736] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5054.349561] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5056.577856] Lustre: Failing over lustre-MDT0000
[ 5056.733723] LustreError: lustre-MDT0000-lwp-OST0000: operation quota_acquire to node 0@lo failed: rc = -107
[ 5056.744159] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5056.764054] Lustre: Skipped 1 previous similar message
[ 5056.771116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5057.053890] Lustre: server umount lustre-MDT0000 complete
[ 5058.532879] LustreError: 6542:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5058.551620] LustreError: 6542:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message
[ 5076.447180] Lustre: 3646:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787392639/real 1787392639]  req@ffff99f387d6d180 x1874211799882880/t0(0) o400->MGC192.168.203.141@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787392655 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5076.480338] Lustre: 3646:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5076.488366] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5078.720855] LDISKFS-fs (dm-0): 7 truncates cleaned up
[ 5078.723753] LDISKFS-fs (dm-0): recovery complete
[ 5078.737622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5087.043966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5087.095688] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5087.826813] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 5091.311836] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5092.335861] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 5092.349522] Lustre: Skipped 1 previous similar message
[ 5092.476896] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted.
[ 5092.538295] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:534 to 0x280000401:577)
[ 5092.539115] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:531 to 0x2c0000401:577)
[ 5100.529512] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5102.478731] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5108.249567] Lustre: DEBUG MARKER: (dd_pid=127847, time=0, timeout=600)
[ 5141.106869] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5147.530880] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 5155.959740] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5157.698231] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5159.553086] Lustre: Failing over lustre-MDT0000
[ 5159.822273] Lustre: server umount lustre-MDT0000 complete
[ 5161.952685] LustreError: lustre-MDT0000-lwp-OST0001: operation ldlm_enqueue to node 0@lo failed: rc = -107
[ 5161.962701] LustreError: Skipped 1 previous similar message
[ 5161.964760] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5161.967381] LustreError: 51946:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5161.967393] LustreError: 51946:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 34 previous similar messages
[ 5162.007332] Lustre: Skipped 4 previous similar messages
[ 5179.360518] Lustre: 3644:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787392742/real 1787392742]  req@ffff99f38ff45c00 x1874211799950976/t0(0) o400->MGC192.168.203.141@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787392758 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5179.397988] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5183.319920] LDISKFS-fs (dm-0): 3 truncates cleaned up
[ 5183.324295] LDISKFS-fs (dm-0): recovery complete
[ 5183.333945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5189.887115] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5189.933891] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5190.239977] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 5194.761657] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5195.242624] LustreError: 3642:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-lwp-MDT0001: namespace resource [0x200000006:0x10000:0x0].0x0 (ffff99f4b5deb400) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5195.272421] LustreError: 3642:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 5 previous similar messages
[ 5195.293744] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo)
[ 5195.299717] Lustre: Skipped 3 previous similar messages
[ 5195.328440] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted.
[ 5195.382733] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:579 to 0x280000401:609)
[ 5195.382796] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:531 to 0x2c0000401:609)
[ 5204.574911] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5206.647437] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5212.319171] Lustre: DEBUG MARKER: (dd_pid=130412, time=0, timeout=600)
[ 5250.580488] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 06:00:28 (1787392828)
[ 5300.486874] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 06:01:18 (1787392878)
[ 5325.656762] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 5336.993682] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 5338.632877] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 5340.435717] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 5342.490579] Lustre: DEBUG MARKER: Set quota for 1 times
[ 5345.828960] Lustre: DEBUG MARKER: Set quota for 2 times
[ 5349.121402] Lustre: DEBUG MARKER: Set quota for 3 times
[ 5352.363176] Lustre: DEBUG MARKER: Set quota for 4 times
[ 5355.982482] Lustre: DEBUG MARKER: Set quota for 5 times
[ 5359.148974] Lustre: DEBUG MARKER: Set quota for 6 times
[ 5362.395194] Lustre: DEBUG MARKER: Set quota for 7 times
[ 5365.326833] Lustre: DEBUG MARKER: Set quota for 8 times
[ 5369.204636] Lustre: DEBUG MARKER: Set quota for 9 times
[ 5411.568224] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 06:03:09 (1787392989)
[ 5430.756656] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5430.763386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5430.766499] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 5430.773032] Lustre: Skipped 5 previous similar messages
[ 5430.790047] Lustre: Skipped 3 previous similar messages
[ 5432.717898] Lustre: server umount lustre-MDT0000 complete
[ 5435.873610] LustreError: 10115:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5435.904808] LustreError: 10115:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 34 previous similar messages
[ 5436.740413] LustreError: 16477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393015 with bad export cookie 6288710491415309518
[ 5436.745279] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5436.753808] LustreError: 16477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 5437.029169] Lustre: server umount lustre-MDT0001 complete
[ 5450.348732] Lustre: server umount lustre-OST0000 complete
[ 5465.163103] Lustre: server umount lustre-OST0001 complete
[ 5481.841357] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local
[ 5491.677854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5492.251674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5496.561746] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5505.467761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5505.834755] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 5510.882470] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5513.198550] Lustre: 154637:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 5520.827602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 5521.130544] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 5528.844743] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5537.795085] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:612 to 0x280000401:641)
[ 5538.414929] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 5538.621914] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 5543.912288] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:611 to 0x2c0000401:641)
[ 5543.920687] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:97)
[ 5544.725956] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5552.675568] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 5556.472701] Lustre: 156513:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 5577.701885] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 5577.719390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5579.746209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5579.750189] Lustre: Skipped 1 previous similar message
[ 5581.370265] Lustre: server umount lustre-MDT0000 complete
[ 5584.863953] LustreError: 153483:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5584.874663] LustreError: 153483:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages
[ 5585.348025] LustreError: 153464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393164 with bad export cookie 6288710491415316728
[ 5585.354531] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5585.367434] LustreError: 153464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 5585.779157] Lustre: server umount lustre-MDT0001 complete
[ 5599.302640] Lustre: server umount lustre-OST0000 complete
[ 5613.489846] Lustre: server umount lustre-OST0001 complete
[ 5629.254504] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local
[ 5639.292688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5639.781486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5644.094635] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5652.687516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5657.417984] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5660.264465] Lustre: 160253:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 5667.231796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 5675.176865] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5684.216435] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:612 to 0x280000401:673)
[ 5685.316630] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 5685.460710] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 5685.464444] Lustre: Skipped 2 previous similar messages
[ 5690.880360] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:611 to 0x2c0000401:673)
[ 5690.887510] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:129)
[ 5691.193473] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5699.858388] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 5703.732625] Lustre: 162122:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 5714.612108] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 06:08:12 (1787393292)
[ 5717.041824] Lustre: DEBUG MARKER: OST0_SIZE: 3605396 required: 6144
[ 5718.843614] Lustre: DEBUG MARKER: run for 4MB test file
[ 5730.995273] Lustre: DEBUG MARKER: User quota (limit: 4 MB)
[ 5737.331578] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT
[ 5739.166387] Lustre: DEBUG MARKER: Write half of file
[ 5741.106713] Lustre: DEBUG MARKER: Write out of block quota ...
[ 5743.060605] Lustre: DEBUG MARKER: Step1: done
[ 5744.627592] Lustre: DEBUG MARKER: Step2: rewrite should succeed
[ 5746.237701] Lustre: DEBUG MARKER: Step2: done
[ 5771.325383] Lustre: DEBUG MARKER: OST0_SIZE: 3605396 required: 61440
[ 5772.913079] Lustre: DEBUG MARKER: run for 40MB test file
[ 5784.648069] Lustre: DEBUG MARKER: User quota (limit: 40 MB)
[ 5790.755720] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT
[ 5792.286779] Lustre: DEBUG MARKER: Write half of file
[ 5795.242887] Lustre: DEBUG MARKER: Write out of block quota ...
[ 5798.874696] Lustre: DEBUG MARKER: Step1: done
[ 5800.456607] Lustre: DEBUG MARKER: Step2: rewrite should succeed
[ 5802.293174] Lustre: DEBUG MARKER: Step2: done
[ 5850.170531] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 06:10:27 (1787393427)
[ 5890.889733] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 06:11:08 (1787393468)
[ 5929.582642] Lustre: DEBUG MARKER: Write...
[ 5931.462659] Lustre: DEBUG MARKER: Write out of block quota ...
[ 5983.639221] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 06:12:41 (1787393561)
[ 5989.910578] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 06:12:47 (1787393567)
[ 6000.617398] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 06:12:58 (1787393578)
[ 6009.880189] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 06:13:07 (1787393587)
[ 6018.745662] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 06:13:16 (1787393596)
[ 6069.274516] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 6185.000525] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 6319.416532] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 06:18:17 (1787393897)
[ 6368.914701] Lustre: DEBUG MARKER: Restart...
[ 6374.367802] LustreError: 3645:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED  req@ffff99f4bf86a680 x1874211800679808/t0(0) o601->lustre-MDT0000-lwp-MDT0000@0@lo:23/10 lens 336/336 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 6374.374883] LustreError: lustre-MDT0000-lwp-MDT0001: operation quota_acquire to node 0@lo failed: rc = -107
[ 6374.375863] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6374.375869] Lustre: Skipped 4 previous similar messages
[ 6374.377900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6374.401861] LustreError: 3645:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8  qsd:lustre-MDT0000 qtype:usr lqe: ffff99f4b427b800 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 2600 qunit:0 qtune:0 edquot:0 default:no revoke:0
[ 6374.443445] LustreError: Skipped 2 previous similar messages
[ 6374.881068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6374.893927] Lustre: Skipped 2 previous similar messages
[ 6376.933394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6376.954617] Lustre: Skipped 3 previous similar messages
[ 6379.367819] Lustre: server umount lustre-MDT0000 complete
[ 6382.051806] LustreError: 171981:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6382.067296] LustreError: 171981:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages
[ 6382.938764] LustreError: 164671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393961 with bad export cookie 6288710491415319038
[ 6382.947997] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6382.949249] LustreError: 164671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 6383.482842] Lustre: server umount lustre-MDT0001 complete
[ 6397.785856] Lustre: server umount lustre-OST0000 complete
[ 6411.757038] Lustre: server umount lustre-OST0001 complete
[ 6427.948642] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local
[ 6438.631276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 6439.230482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6443.566677] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6453.033938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 6453.469246] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 6458.850063] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6462.993340] Lustre: 186423:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6472.035246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 6472.591904] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 6479.802256] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6484.988977] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:685 to 0x280000401:705)
[ 6488.879325] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 6494.194882] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:681 to 0x2c0000401:705)
[ 6494.204617] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:161)
[ 6496.157863] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6503.959188] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6508.358686] Lustre: 188295:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6570.758170] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 06:22:28 (1787394148)
[ 6623.113333] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 06:23:20 (1787394200)
[ 8418.331553] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 06:53:15 (1787395995)
[ 8429.539242] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 8429.550803] LustreError: Skipped 1 previous similar message
[ 8429.557452] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 8429.571388] Lustre: Skipped 3 previous similar messages
[ 8429.580937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 8432.616469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 8432.636575] Lustre: Skipped 3 previous similar messages
[ 8437.728658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 8437.737255] Lustre: Skipped 3 previous similar messages
[ 8442.849805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 8442.866085] Lustre: Skipped 3 previous similar messages
[ 8443.359150] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck?
[ 8445.799111] Lustre: server umount lustre-MDT0000 complete
[ 8447.969832] LustreError: 185280:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 8447.994332] LustreError: 185280:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages
[ 8449.863669] LustreError: 188298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396028 with bad export cookie 6288710491415328145
[ 8449.864981] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 8449.876407] LustreError: 188298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message
[ 8450.216107] Lustre: server umount lustre-MDT0001 complete
[ 8464.654505] Lustre: server umount lustre-OST0000 complete
[ 8478.803506] Lustre: server umount lustre-OST0001 complete
[ 8494.421587] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local
[ 8504.244126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 8504.742339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 8504.748139] Lustre: Skipped 1 previous similar message
[ 8508.463892] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8515.040262] LustreError: 195896:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 8515.056809] LustreError: 195896:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages
[ 8515.184729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 8515.615860] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 8519.211242] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8521.771381] Lustre: 197039:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 8528.494675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 8528.829716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 8529.851457] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5707 to 0x280000401:5729)
[ 8534.508547] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8544.234656] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 8549.873634] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5707 to 0x2c0000401:5729)
[ 8549.930568] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:193)
[ 8551.170793] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8558.183963] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 8561.258592] Lustre: 198909:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 8585.509628] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 06:56:03 (1787396163)
[ 8610.688518] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 06:56:28 (1787396188)
[ 8634.189321] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 06:56:52 (1787396212)
[ 8667.276994] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 06:57:25 (1787396245)
[ 8705.736311] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 06:58:03 (1787396283)
[ 8779.132569] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 06:59:17 (1787396357)
[ 8810.313610] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 06:59:47 (1787396387)
[ 8872.775961] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 07:00:50 (1787396450)
[ 8877.739520] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8877.744091] Lustre: Skipped 1 previous similar message
[ 8879.747732] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8879.751919] Lustre: Skipped 87 previous similar messages
[ 8883.784722] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8883.786841] Lustre: Skipped 167 previous similar messages
[ 8891.800859] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8891.804376] Lustre: Skipped 351 previous similar messages
[ 8907.830052] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8907.834526] Lustre: Skipped 703 previous similar messages
[ 8939.848882] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8939.851760] Lustre: Skipped 1397 previous similar messages
[ 9003.885401] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9003.888739] Lustre: Skipped 2647 previous similar messages
[ 9333.858154] Lustre: *** cfs_fail_loc=a08, val=0***
[ 9333.865593] Lustre: Skipped 2644 previous similar messages
[ 9333.867811] Lustre: *** cfs_fail_loc=a08, val=0***
[ 9333.883945] Lustre: Skipped 3 previous similar messages
[ 9415.140376] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 9415.142874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 9415.163422] Lustre: Skipped 5 previous similar messages
[ 9415.172846] Lustre: Skipped 3 previous similar messages
[ 9420.256881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 9420.266820] Lustre: Skipped 3 previous similar messages
[ 9421.160217] Lustre: server umount lustre-MDT0000 complete
[ 9424.653592] LustreError: 204544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397003 with bad export cookie 6288710491417109064
[ 9424.670482] LustreError: 204544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message
[ 9424.671440] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 9425.377764] LustreError: 206384:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9425.382337] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 9425.387387] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping)
[ 9425.400470] LustreError: 206384:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages
[ 9425.410069] Lustre: Skipped 2 previous similar messages
[ 9429.413347] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping)
[ 9429.429886] Lustre: Skipped 2 previous similar messages
[ 9431.016753] Lustre: server umount lustre-MDT0001 complete
[ 9434.851985] Lustre: server umount lustre-OST0000 complete
[ 9439.374745] Lustre: server umount lustre-OST0001 complete
[ 9446.325318] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_hostid
[ 9455.066034] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local
[ 9498.750623] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local
[ 9508.299377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 9508.556238] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000
[ 9508.603488] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[ 9508.683761] Lustre: lustre-MDT0000: new disk, initializing
[ 9508.761241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 9508.766064] Lustre: Skipped 1 previous similar message
[ 9508.780862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[ 9512.922221] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9522.363149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 9522.468172] Lustre: 215475:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17
[ 9522.496611] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
[ 9522.502404] Lustre: Skipped 1 previous similar message
[ 9522.578888] Lustre: lustre-MDT0001: new disk, initializing
[ 9522.657626] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 9522.707011] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt
[ 9522.725266] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt]
[ 9527.376459] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9532.428367] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params
[ 9538.637773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 9538.907565] Lustre: lustre-OST0000: new disk, initializing
[ 9538.912608] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[ 9538.922679] Lustre: 217105:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0]
[ 9538.991113] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 9540.519236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost
[ 9540.533929] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost]
[ 9540.627919] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401
[ 9544.145515] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9555.562043] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 9555.722770] Lustre: lustre-OST0001: new disk, initializing
[ 9555.731504] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[ 9555.742971] Lustre: 217976:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0]
[ 9555.820160] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 9557.401829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost
[ 9557.411472] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost]
[ 9557.473589] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401
[ 9562.720623] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9574.350330] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 9578.469324] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params
[ 9605.802626] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 07:13:03 (1787397183)
[ 9612.475701] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 07:13:10 (1787397190)
[ 9646.053538] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 07:13:43 (1787397223)
[ 9697.382926] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 07:14:35 (1787397275)
[ 9720.135597] Lustre: DEBUG MARKER: rename directory return 255
[ 9757.741974] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 07:15:35 (1787397335)
[ 9782.719736] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 07:16:00 (1787397360)
[ 9820.755736] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 07:16:38 (1787397398)
[ 9948.566048] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 07:18:46 (1787397526)
[ 9976.718373] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 07:19:14 (1787397554)
[10011.879654] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 07:19:49 (1787397589)
[10149.833189] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 07:22:07 (1787397727)
[10154.464512] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[10154.480536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[10154.489916] Lustre: Skipped 3 previous similar messages
[10159.585306] LustreError: 215486:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[10159.605531] LustreError: 215486:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages
[10159.682713] Lustre: server umount lustre-MDT0000 complete
[10162.757173] LustreError: 217975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397741 with bad export cookie 6288710491417481254
[10162.761857] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[10162.777622] LustreError: 217975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages
[10163.102415] Lustre: server umount lustre-MDT0001 complete
[10177.161547] Lustre: server umount lustre-OST0000 complete
[10180.063154] Lustre: 3643:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787397743/real 1787397743]  req@ffff99f399984e00 x1874211811311232/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787397759 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[10180.085703] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[10180.096741] Lustre: Skipped 3 previous similar messages
[10180.722968] Lustre: server umount lustre-OST0001 complete
[10199.115751] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local
[10207.830702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[10208.252601] LustreError: 234973:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[10208.329848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[10208.347600] LustreError: lustre-MDT0000: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[10212.150704] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10213.349613] LustreError: 234974:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[10220.807948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[10221.078703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[10221.090893] LustreError: lustre-MDT0001: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[10225.814649] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10228.346795] Lustre: 236118:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[10233.979373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[10234.245694] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[10234.251833] LustreError: lustre-OST0000: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[10240.193205] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10245.540774] LustreError: 236474:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[10245.565217] LustreError: 236474:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message
[10247.978979] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[10248.207446] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[10248.216421] LustreError: lustre-OST0001: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[10254.067249] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10254.326432] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:20 to 0x280000401:65)
[10254.343891] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:20 to 0x2c0000401:65)
[10261.001459] Lustre: DEBUG MARKER: Using TIMEOUT=20
[10264.618750] Lustre: 237993:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[10271.287640] LustreError: 234969:0:(osd_handler.c:3426:osd_quota_transfer()) lustre-MDT0000: quota transfer failed. Is project enforcement enabled on the ldiskfs filesystem? rc = -95
[10277.344110] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[10277.348268] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[10277.362522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[10279.906401] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[10279.908381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[10279.924352] Lustre: Skipped 2 previous similar messages
[10279.939460] Lustre: Skipped 3 previous similar messages
[10281.474180] Lustre: server umount lustre-MDT0000 complete
[10285.025082] LustreError: 234968:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[10285.043415] LustreError: 234968:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages
[10285.371664] LustreError: 235718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397864 with bad export cookie 6288710491417530079
[10285.371861] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[10285.385193] LustreError: 235718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages
[10285.718349] Lustre: server umount lustre-MDT0001 complete
[10298.335927] LustreError: 3644:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED  req@ffff99f38a05ad80 x1874211811365248/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[10298.366105] LustreError: 3645:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8  qsd:lustre-OST0000 qtype:usr lqe: ffff99f383e0b080 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 1516 qunit:0 qtune:0 edquot:0 default:no revoke:0
[10298.391806] LustreError: 3645:0:(qsd_handler.c:298:qsd_req_completion()) Skipped 1 previous similar message
[10300.451524] Lustre: server umount lustre-OST0000 complete
[10310.623582] LustreError: 3645:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED  req@ffff99f3820eb480 x1874211811365632/t0(0) o601->lustre-MDT0000-lwp-OST0001@0@lo:23/10 lens 336/336 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[10310.650033] LustreError: 3645:0:(client.c:1381:ptlrpc_import_delay_req()) Skipped 1 previous similar message
[10310.657317] LustreError: 3645:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8  qsd:lustre-OST0001 qtype:usr lqe: ffff99f383e09980 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 1516 qunit:0 qtune:0 edquot:0 default:no revoke:0
[10310.686581] LustreError: 3645:0:(qsd_handler.c:298:qsd_req_completion()) Skipped 1 previous similar message
[10314.042753] Lustre: server umount lustre-OST0001 complete
[10332.855453] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local
[10344.984985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[10345.538618] LustreError: 240382:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[10345.558210] LustreError: 240382:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message
[10345.631484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[10351.144960] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10359.352634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[10362.963647] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10365.920928] Lustre: 241529:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[10372.866874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[10379.641399] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10387.653614] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[10387.844625] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[10387.849691] Lustre: Skipped 2 previous similar messages
[10390.960557] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97)
[10390.994310] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:20 to 0x2c0000401:97)
[10393.277197] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10400.647256] Lustre: DEBUG MARKER: Using TIMEOUT=20
[10404.353915] Lustre: 243407:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[10433.575495] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 07:26:51 (1787398011)
[10484.821232] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[10487.289545] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 07:27:44 (1787398064)
[10507.429863] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[10509.027096] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 07:28:06 (1787398086)
[10550.671588] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[10552.746968] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 07:28:50 (1787398130)
[10592.124655] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 07:29:29 (1787398169)
[10603.755301] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[10610.052036] Lustre: 250262:0:(qsd_reint.c:249:qsd_reint_index()) lustre-OST0001: index version for fid [0x200000005:0x100a:0x0] is 0, but index isn't empty (1)
[10610.076500] Lustre: 250262:0:(qsd_reint.c:249:qsd_reint_index()) Skipped 2 previous similar messages
[10615.263384] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[10621.157069] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[10625.439077] Lustre: DEBUG MARKER: Write...
[10645.785913] LustreError: 251604:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[10651.587379] Lustre: DEBUG MARKER: Write...
[10665.170786] Lustre: DEBUG MARKER: Write...
[10740.090326] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 07:31:57 (1787398317)
[10764.365217] LustreError: 255715:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[10807.203956] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 07:33:04 (1787398384)
[10835.338110] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[10836.906423] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[10925.871455] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 07:35:03 (1787398503)
[10980.154815] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 07:35:57 (1787398557)
[11017.439455] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 07:36:34 (1787398594)
[11029.430868] Lustre: DEBUG MARKER: User quota (block hardlimit:100 MB)
[11051.438931] Lustre: DEBUG MARKER: Write...
[11053.583792] Lustre: DEBUG MARKER: Write out of block quota ...
[11149.183740] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 07:38:47 (1787398727)
[11160.949929] Lustre: DEBUG MARKER: User quota (block hardlimit:1000 MB)
[11187.117230] LustreError: 240381:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool2 id:60000 enforced:1 hard:10240 soft:0 granted:10240 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[11246.359258] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 07:40:24 (1787398824)
[11258.150492] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB)
[11279.268458] Lustre: DEBUG MARKER: Write...
[11282.032491] Lustre: DEBUG MARKER: Write out of block quota ...
[11338.741609] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 07:41:56 (1787398916)
[11366.181414] Lustre: DEBUG MARKER: set to use default quota
[11367.868791] Lustre: DEBUG MARKER: set default quota
[11369.382506] Lustre: DEBUG MARKER: get default quota
[11374.660085] Lustre: DEBUG MARKER: Test not out of quota
[11377.873137] Lustre: DEBUG MARKER: Test out of quota
[11388.327578] Lustre: DEBUG MARKER: Increase default quota
[11409.980454] Lustre: DEBUG MARKER: Set quota to override default quota
[11410.032761] LustreError: 247069:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45056 time:1788003788 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[11420.978382] Lustre: DEBUG MARKER: Set to use default quota again
[11440.573257] Lustre: DEBUG MARKER: Cleanup
[11499.605639] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 07:44:37 (1787399077)
[11522.968119] Lustre: DEBUG MARKER: set default quota for qpool1
[11524.787399] Lustre: DEBUG MARKER: Write from user that hasn't lqe
[11575.776179] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 07:45:52 (1787399152)
[11662.083619] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 07:47:19 (1787399239)
[11728.537815] Lustre: DEBUG MARKER: Write...
[11731.747837] Lustre: DEBUG MARKER: Write out of block quota ...
[11842.328558] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 07:50:19 (1787399419)
[11875.985572] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 07:50:53 (1787399453)
[11884.748791] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 07:51:02 (1787399462)
[11929.280427] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 07:51:46 (1787399506)
[11975.644622] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 07:52:33 (1787399553)
[12001.115210] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 07:52:58 (1787399578)
[12032.638025] Lustre: *** cfs_fail_loc=a06, val=0***
[12032.646907] Lustre: Skipped 4 previous similar messages
[12033.186102] LustreError: 240396:0:(qmt_lock.c:479:qmt_lvbo_update()) $$$ failed to release quota space on glimpse 0!=1024 : rc = -11
                 qmt:lustre-QMT0000 pool:dt-0x0 id:60000 enforced:1 hard:102400 soft:0 granted:24576 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no
[12038.354780] Lustre: Failing over lustre-OST0001
[12038.430615] Lustre: server umount lustre-OST0001 complete
[12039.655529] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[12039.658645] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107
[12039.678725] LustreError: Skipped 1 previous similar message
[12039.680389] LustreError: 241884:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[12039.696705] LustreError: 241884:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages
[12049.010276] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[12049.348504] Lustre: lustre-OST0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180
[12049.367461] Lustre: lustre-OST0001: in recovery but waiting for the first client to connect
[12051.000490] Lustre: lustre-OST0001: Will be in recovery for at least 1:00, or until 3 clients reconnect
[12051.081184] Lustre: lustre-OST0001: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted.
[12051.084212] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo)
[12051.107719] Lustre: Skipped 3 previous similar messages
[12056.851353] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[12109.225100] Lustre: DEBUG MARKER: == sanity-quota test 81: Race qmt_start_pool_recalc with qmt_pool_free ========================================================== 07:54:46 (1787399686)
[12122.617211] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[12131.662482] LustreError: 294944:0:(qmt_pool.c:1413:qmt_pool_recalc()) cfs_fail_timeout id a07 sleeping for 10000ms
[12136.433371] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[12136.437608] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[12136.441434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[12136.503286] Lustre: Skipped 4 previous similar messages
[12138.082248] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.41@tcp (stopping)
[12138.090687] Lustre: Skipped 3 previous similar messages
[12141.544688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[12141.556583] Lustre: Skipped 3 previous similar messages
[12141.663346] LustreError: 294944:0:(qmt_pool.c:1413:qmt_pool_recalc()) cfs_fail_timeout id a07 awake
[12142.033466] Lustre: server umount lustre-MDT0000 complete
[12143.187753] LustreError: 240377:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[12143.220918] LustreError: 240377:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages
[12151.353523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[12151.542792] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[12151.864492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[12151.909568] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:112 to 0x280000401:129)
[12151.919713] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:113 to 0x2c0000401:129)
[12156.434272] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[12156.922618] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo)
[12156.932494] Lustre: Skipped 1 previous similar message
[12191.511634] Lustre: DEBUG MARKER: == sanity-quota test 82: verify more than 8 qids for single operation ========================================================== 07:56:08 (1787399768)
[12200.010322] LustreError: 241131:0:(qsd_reint.c:633:qqi_reint_delayed()) lustre-MDT0001: Delaying reintegration for qtype:2 until pending updates are flushed.
[12221.938203] Lustre: DEBUG MARKER: == sanity-quota test 83: Setting default quota shouldn't affect grace time ========================================================== 07:56:38 (1787399798)
[12245.003598] Lustre: DEBUG MARKER: == sanity-quota test 84: Reset quota should fix the insane granted quota ========================================================== 07:57:02 (1787399822)
[12295.688309] Lustre: *** cfs_fail_loc=a08, val=0***
[12295.700083] Lustre: Skipped 1432 previous similar messages
[12295.708068] Lustre: *** cfs_fail_loc=a08, val=0***
[12295.715074] Lustre: Skipped 11 previous similar messages
[12393.947036] Lustre: DEBUG MARKER: == sanity-quota test 85: do not hung at write with the least_qunit ========================================================== 07:59:31 (1787399971)
[12424.618899] LustreError: 240380:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:3072 soft:0 granted:3072 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[12482.705457] Lustre: DEBUG MARKER: == sanity-quota test 86: Pre-acquired quota should be released if quota is over limit ========================================================== 08:00:59 (1787400059)
[12555.038897] LustreError: 241492:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:16384 time:1788004933 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[12717.564490] LustreError: 240378:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:16384 time:1788005096 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[12726.570601] Lustre: DEBUG MARKER: sanity-quota test_86: @@@@@@ FAIL: succeed to create files, expect failed