[ 5315.830336] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5322.895200] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 5325.813655] LustreError: 123076:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5326.938641] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5328.343256] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5330.743928] Lustre: Failing over lustre-MDT0000
[ 5331.109647] Lustre: server umount lustre-MDT0000 complete
[ 5348.429162] LustreError: MGC192.168.206.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5348.816365] Lustre: 3310:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789140923/real 1789140923]  req@ffff9ab48f06e1c0 x1876044742353792/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1789140939 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 5348.848439] Lustre: 3310:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 5348.854492] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5348.868992] Lustre: Skipped 1 previous similar message
[ 5349.123381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5349.196246] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5350.893608] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5351.075890] Lustre: 3309:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789140925/real 1789140925]  req@ffff9ab48f06e580 x1876044742354176/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789140941 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5351.111179] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5351.166890] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:654 to 0x280000400:673)
[ 5351.169906] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:658 to 0x240000400:673)
[ 5352.963468] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5354.473991] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 5355.488223] Lustre: 3308:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789140930/real 1789140930]  req@ffff9ab5934d0000 x1876044742354432/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789140946 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5355.521163] Lustre: 3308:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5360.608148] Lustre: 3310:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789140935/real 1789140935]  req@ffff9ab5934d0b40 x1876044742354816/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789140951 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5360.664371] Lustre: 3310:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5361.977086] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5363.673967] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5370.080721] Lustre: DEBUG MARKER: (dd_pid=115450, time=1, timeout=600)
[ 5415.407242] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5421.579520] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 5424.770108] LustreError: 125934:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5425.660825] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5427.163746] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5429.096659] Lustre: Failing over lustre-MDT0000
[ 5429.337988] Lustre: server umount lustre-MDT0000 complete
[ 5446.627328] Lustre: 3308:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789141021/real 1789141021]  req@ffff9ab595f60780 x1876044742389888/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1789141037 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 5446.661697] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5447.712790] LustreError: MGC192.168.206.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5457.889371] LustreError: 3306:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight  req@ffff9ab48ed8c780 x1876044742393216/t0(0) o250->MGC192.168.206.153@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5458.486383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5458.629426] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5459.409524] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5459.646211] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5459.681881] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:654 to 0x280000400:705)
[ 5459.683535] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:675 to 0x240000400:705)
[ 5462.813297] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5471.735616] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[ 5471.750209] Lustre: Skipped 1 previous similar message
[ 5472.567505] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5472.807133] LustreError: 3306:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-lwp-OST0001: namespace resource [0x200000006:0x20000:0x0].0x0 (ffff9ab4c8f73600) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5472.843753] LustreError: 3306:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 5 previous similar messages
[ 5474.120792] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5494.807710] Lustre: DEBUG MARKER: (dd_pid=117845, time=11, timeout=600)
[ 5562.594055] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 11:39:11 (1789141151)
[ 5587.405432] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 5589.705809] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 5595.459730] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 5597.145292] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 5635.411926] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 11:40:24 (1789141224)
[ 5662.339151] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 5680.201335] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 5682.243499] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 5684.795568] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 5687.267876] Lustre: DEBUG MARKER: Set quota for 1 times
[ 5691.663626] Lustre: DEBUG MARKER: Set quota for 2 times
[ 5695.415687] Lustre: DEBUG MARKER: Set quota for 3 times
[ 5699.042769] Lustre: DEBUG MARKER: Set quota for 4 times
[ 5702.542826] Lustre: DEBUG MARKER: Set quota for 5 times
[ 5705.751078] Lustre: DEBUG MARKER: Set quota for 6 times
[ 5709.224234] Lustre: DEBUG MARKER: Set quota for 7 times
[ 5712.408570] Lustre: DEBUG MARKER: Set quota for 8 times
[ 5715.887462] Lustre: DEBUG MARKER: Set quota for 9 times
[ 5772.466214] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 11:42:41 (1789141361)
[ 5785.069595] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5785.077272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5785.085083] Lustre: Skipped 2 previous similar messages
[ 5785.099076] Lustre: Skipped 1 previous similar message
[ 5786.354840] Lustre: server umount lustre-MDT0000 complete
[ 5790.429831] LustreError: 5821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789141381 with bad export cookie 10589606894650574467
[ 5790.431314] LustreError: MGC192.168.206.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5790.441104] LustreError: 5821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message
[ 5790.564709] Lustre: server umount lustre-OST0000 complete
[ 5794.960383] Lustre: server umount lustre-OST0001 complete
[ 5809.687211] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing load_modules_local
[ 5819.884556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5824.846278] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5827.934535] Lustre: 135878:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 5834.549548] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 5840.814680] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5844.713593] LustreError: 136248:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5844.744293] LustreError: 136248:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message
[ 5844.753089] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:707 to 0x240000400:737)
[ 5848.761080] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 5854.211212] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:708 to 0x280000400:737)
[ 5854.975935] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5861.912493] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 5865.608145] Lustre: 137804:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 5884.542067] Lustre: server umount lustre-MDT0000 complete
[ 5889.491564] LustreError: 135425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789141480 with bad export cookie 10589606894650581698
[ 5889.511609] LustreError: MGC192.168.206.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5899.893772] Lustre: server umount lustre-OST0000 complete
[ 5900.257179] Lustre: 3310:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789141475/real 1789141475]  req@ffff9ab587883c00 x1876044742562688/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789141491 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5900.299572] Lustre: 3310:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 10 previous similar messages
[ 5904.429342] Lustre: server umount lustre-OST0001 complete
[ 5919.374437] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing load_modules_local
[ 5927.836220] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5933.222580] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5936.394381] Lustre: 140458:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 5943.912045] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 5952.655606] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5953.058822] LustreError: 140825:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5953.083582] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:707 to 0x240000400:769)
[ 5958.115600] LustreError: 140825:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5966.360663] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:708 to 0x280000400:769)
[ 5966.786963] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5975.430609] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 5979.985631] Lustre: 142363:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 5994.891563] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 11:46:23 (1789141583)
[ 5997.206525] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 5999.718714] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 11:46:28 (1789141588)
[ 6062.049815] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 11:47:31 (1789141651)
[ 6114.811871] Lustre: DEBUG MARKER: Write...
[ 6117.349893] Lustre: DEBUG MARKER: Write out of block quota ...
[ 6173.081545] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 11:49:22 (1789141762)
[ 6181.222509] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 11:49:29 (1789141769)
[ 6192.305635] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 11:49:41 (1789141781)
[ 6202.169240] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 11:49:51 (1789141791)
[ 6210.254960] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 11:49:59 (1789141799)
[ 6275.116283] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 6430.636871] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 6617.849688] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 11:56:46 (1789142206)
[ 6684.686461] Lustre: DEBUG MARKER: Restart...
[ 6693.351981] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6693.356605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6693.378951] Lustre: Skipped 2 previous similar messages
[ 6693.414499] Lustre: Skipped 1 previous similar message
[ 6696.064760] Lustre: server umount lustre-MDT0000 complete
[ 6700.509851] LustreError: 144586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789142291 with bad export cookie 10589606894650583224
[ 6700.512337] LustreError: MGC192.168.206.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6700.525392] LustreError: 144586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message
[ 6700.671911] Lustre: server umount lustre-OST0000 complete
[ 6704.958695] Lustre: server umount lustre-OST0001 complete
[ 6721.037460] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing load_modules_local
[ 6730.659286] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6730.664731] Lustre: Skipped 1 previous similar message
[ 6734.918093] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6737.569417] Lustre: 160891:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6744.495246] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 6751.003437] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6755.683335] LustreError: 161251:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6755.731747] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:778 to 0x240000400:801)
[ 6759.545618] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6764.549222] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:778 to 0x280000400:801)
[ 6766.056710] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6772.350384] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6775.539452] Lustre: 162807:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6841.718468] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 12:00:30 (1789142430)
[ 6918.227985] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 12:01:46 (1789142506)
[ 8723.843725] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 12:31:52 (1789144312)
[ 8742.334766] Lustre: server umount lustre-MDT0000 complete
[ 8746.734443] LustreError: 167447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789144337 with bad export cookie 10589606894650589727
[ 8746.743721] LustreError: MGC192.168.206.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 8756.400330] Lustre: server umount lustre-OST0000 complete
[ 8759.265538] Lustre: 3307:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789144334/real 1789144334]  req@ffff9ab4ca00bc00 x1876044745765376/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789144350 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 8759.291130] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 8760.962752] Lustre: server umount lustre-OST0001 complete
[ 8775.492523] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing load_modules_local
[ 8785.750319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 8790.512231] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8794.157385] Lustre: 171352:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 8801.064570] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 8807.939704] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8810.212626] LustreError: 171721:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 8810.248351] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5803 to 0x240000400:5825)
[ 8815.592604] LustreError: 171721:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 8815.863838] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 8821.289454] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5803 to 0x280000400:5825)
[ 8823.297830] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8832.612909] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 8837.381273] Lustre: 173263:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 8873.110483] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 12:34:22 (1789144462)
[ 8918.534469] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 12:35:07 (1789144507)
[ 8964.026414] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 12:35:52 (1789144552)
[ 8966.492239] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs
[ 8969.019581] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 12:35:57 (1789144557)
[ 8970.734877] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs
[ 8972.788922] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 12:36:01 (1789144561)
[ 9055.905246] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 12:37:25 (1789144645)
[ 9106.080917] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 12:38:14 (1789144694)
[ 9197.758823] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 12:39:46 (1789144786)
[ 9210.407893] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9210.423429] Lustre: Skipped 3 previous similar messages
[ 9212.462443] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9212.471821] Lustre: Skipped 57 previous similar messages
[ 9216.491754] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9216.499119] Lustre: Skipped 179 previous similar messages
[ 9224.523650] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9224.527405] Lustre: Skipped 331 previous similar messages
[ 9240.541472] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9240.546781] Lustre: Skipped 727 previous similar messages
[ 9272.565889] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9272.571394] Lustre: Skipped 1537 previous similar messages
[ 9336.567416] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9336.569650] Lustre: Skipped 2756 previous similar messages
[ 9749.730580] Lustre: *** cfs_fail_loc=a08, val=0***
[ 9749.732405] Lustre: Skipped 2407 previous similar messages
[ 9749.745458] Lustre: *** cfs_fail_loc=a08, val=0***
[ 9749.753292] Lustre: Skipped 1 previous similar message
[10007.518329] Lustre: server umount lustre-MDT0000 complete
[10010.862906] LustreError: 170904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789145601 with bad export cookie 10589606894652369813
[10010.877136] LustreError: 170904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages
[10010.889299] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[10010.902383] Lustre: Skipped 1 previous similar message
[10011.500558] LustreError: 3307:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x4  qsd:lustre-OST0000 qtype:grp lqe: ffff9ab5bb943500 id:61 enforced:1 granted: 16384 pending:0 waiting:0 req:1 usage: 0 qunit:0 qtune:0 edquot:0 default:no revoke:0
[10011.502048] LustreError: 170906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789145602 with bad export cookie 10589606894652369428
[10011.506020] LustreError: 3308:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED  req@ffff9ab4ca258000 x1876044750670848/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[10011.525642] LustreError: 3307:0:(qsd_handler.c:298:qsd_req_completion()) Skipped 30 previous similar messages
[10011.535869] LustreError: 170906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 74 previous similar messages
[10011.580799] LustreError: MGC192.168.206.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[10011.895847] Lustre: server umount lustre-OST0000 complete
[10014.943595] LustreError: 3309:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x4  qsd:lustre-OST0001 qtype:usr lqe: ffff9ab4cf5b8780 id:26 enforced:1 granted: 16384 pending:0 waiting:0 req:1 usage: 0 qunit:0 qtune:0 edquot:0 default:no revoke:0
[10014.957214] LustreError: 3309:0:(qsd_handler.c:298:qsd_req_completion()) Skipped 24 previous similar messages
[10015.536704] Lustre: server umount lustre-OST0001 complete
[10022.360881] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_hostid
[10029.697750] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing load_modules_local
[10037.825313]  vdc: vdc1 vdc9
[10047.022911]  vde: vde1 vde9
[10058.120415]  vdf: vdf1 vdf9
[10075.405145] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing load_modules_local
[10083.770861] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000
[10083.991262] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[10084.047487] Lustre: lustre-MDT0000: new disk, initializing
[10084.459701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[10084.571608] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[10089.861676] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10095.171408] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params
[10102.148054] Lustre: lustre-OST0000: new disk, initializing
[10102.152958] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[10102.164304] Lustre: Skipped 1 previous similar message
[10102.298090] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[10104.402973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost
[10104.418914] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost]
[10104.680281] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400
[10110.704473] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10121.244736] Lustre: lustre-OST0001: new disk, initializing
[10121.249793] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[10121.364105] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[10123.216411] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost
[10123.226821] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost]
[10123.379198] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400
[10128.043604] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10137.527394] Lustre: DEBUG MARKER: Using TIMEOUT=20
[10140.849994] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params
[10178.297497] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 12:56:07 (1789145767)
[10184.942951] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 12:56:14 (1789145774)
[10231.462994] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 12:57:00 (1789145820)
[10293.681241] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 12:58:02 (1789145882)
[10331.451697] Lustre: DEBUG MARKER: rename directory return 255
[10380.561460] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 12:59:29 (1789145969)
[10415.973496] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 13:00:04 (1789146004)
[10461.397731] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 13:00:50 (1789146050)
[10581.605739] LustreError: 198246:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-0x0 id:60001 enforced:1 hard:51200 soft:0 granted:51200 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[10621.956495] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 13:03:30 (1789146210)
[10664.444408] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 13:04:13 (1789146253)
[10706.028277] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 13:04:55 (1789146295)
[10876.617753] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 13:07:45 (1789146465)
[10878.372735] Lustre: DEBUG MARKER: SKIP: sanity-quota test_59 ldiskfs only test
[10879.876674] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 13:07:49 (1789146469)
[10938.365323] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[10940.496692] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 13:08:49 (1789146529)
[10972.098233] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[10973.897799] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 13:09:22 (1789146562)
[11027.087769] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[11028.601314] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 13:10:17 (1789146617)
[11078.910962] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 13:11:07 (1789146667)
[11080.358741] Lustre: DEBUG MARKER: SKIP: sanity-quota test_67 ZFS grants some block space together with inode
[11082.013622] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 13:11:11 (1789146671)
[11107.127316] LustreError: 211223:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[11148.692270] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 13:12:17 (1789146737)
[11174.035397] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[11175.783470] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[11365.260957] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 13:15:54 (1789146954)
[11415.053818] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 13:16:44 (1789147004)
[11452.627994] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 13:17:21 (1789147041)
[11454.556633] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71a ZFS grants some block space together with inode
[11456.658795] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 13:17:25 (1789147045)
[11458.237854] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71b ZFS grants some block space together with inode
[11459.882023] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 13:17:29 (1789147049)
[11475.275042] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB)
[11491.311602] Lustre: DEBUG MARKER: Write...
[11493.649796] Lustre: DEBUG MARKER: Write out of block quota ...
[11545.738405] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 13:18:54 (1789147134)
[11572.074273] Lustre: DEBUG MARKER: set to use default quota
[11573.681249] Lustre: DEBUG MARKER: set default quota
[11575.268913] Lustre: DEBUG MARKER: get default quota
[11581.295061] Lustre: DEBUG MARKER: Test not out of quota
[11586.533556] Lustre: DEBUG MARKER: Test out of quota
[11598.651163] Lustre: DEBUG MARKER: Increase default quota
[11630.626873] Lustre: DEBUG MARKER: Set quota to override default quota
[11630.679688] LustreError: 188160:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45063 time:1789752021 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[11641.652660] Lustre: DEBUG MARKER: Set to use default quota again
[11660.496741] Lustre: DEBUG MARKER: Cleanup
[11729.717292] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 13:21:58 (1789147318)
[11752.850297] Lustre: DEBUG MARKER: set default quota for qpool1
[11754.181838] Lustre: DEBUG MARKER: Write from user that hasn't lqe
[11803.314076] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 13:23:12 (1789147392)
[11878.469714] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 13:24:27 (1789147467)
[11954.153295] Lustre: DEBUG MARKER: Write...
[11957.390715] Lustre: DEBUG MARKER: Write out of block quota ...
[12080.176958] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 13:27:48 (1789147668)
[12127.266139] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 13:28:36 (1789147716)
[12135.110841] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 13:28:43 (1789147723)
[12136.724468] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78A need >= 2.13.57 and ldiskfs for fallocate
[12138.414324] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 13:28:47 (1789147727)
[12140.314542] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78a need >= 2.13.57 and ldiskfs for fallocate
[12142.362359] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 13:28:51 (1789147731)
[12161.607841] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 13:29:10 (1789147750)
[12163.302786] Lustre: DEBUG MARKER: SKIP: sanity-quota test_80 ZFS grants some block space together with inode
[12164.980075] Lustre: DEBUG MARKER: == sanity-quota test 81: Race qmt_start_pool_recalc with qmt_pool_free ========================================================== 13:29:14 (1789147754)
[12181.149427] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[12187.854122] LustreError: 236643:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 sleeping for 10000ms
[12191.714550] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[12191.721202] Lustre: Skipped 1 previous similar message
[12191.722021] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[12191.733280] Lustre: Skipped 1 previous similar message
[12194.268745] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.53@tcp (stopping)
[12196.833791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[12196.846125] Lustre: Skipped 1 previous similar message
[12197.912306] LustreError: 236643:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 awake
[12198.212971] Lustre: server umount lustre-MDT0000 complete
[12205.204984] LustreError: MGC192.168.206.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[12205.741414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[12205.876204] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:27 to 0x280000400:65)
[12205.886739] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:27 to 0x240000400:65)
[12210.528557] Lustre: DEBUG MARKER: oleg653-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[12222.053526] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[12222.067701] Lustre: Skipped 1 previous similar message
[12244.626621] Lustre: DEBUG MARKER: == sanity-quota test 82: verify more than 8 qids for single operation ========================================================== 13:30:33 (1789147833)
[12281.577373] Lustre: DEBUG MARKER: == sanity-quota test 83: Setting default quota shouldn't affect grace time ========================================================== 13:31:10 (1789147870)
[12319.475952] Lustre: DEBUG MARKER: == sanity-quota test 84: Reset quota should fix the insane granted quota ========================================================== 13:31:47 (1789147907)
[12376.102755] Lustre: *** cfs_fail_loc=a08, val=0***
[12376.109807] Lustre: Skipped 13 previous similar messages
[12376.120633] Lustre: *** cfs_fail_loc=a08, val=0***
[12376.135806] Lustre: Skipped 15 previous similar messages
[12502.806788] Lustre: DEBUG MARKER: == sanity-quota test 85: do not hung at write with the least_qunit ========================================================== 13:34:51 (1789148091)
[12591.517832] Lustre: DEBUG MARKER: == sanity-quota test 86: Pre-acquired quota should be released if quota is over limit ========================================================== 13:36:20 (1789148180)
[12677.575793] LustreError: 237228:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1789753068 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[12856.371605] LustreError: 237227:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1789753247 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[12865.545266] Lustre: DEBUG MARKER: sanity-quota test_86: @@@@@@ FAIL: succeed to create files, expect failed