[ 5734.865830] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5742.514462] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 5746.143639] LustreError: 120204:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5747.724834] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5749.837777] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5752.287855] Lustre: Failing over lustre-MDT0000
[ 5752.289787] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5752.340857] Lustre: Skipped 1 previous similar message
[ 5753.023463] Lustre: server umount lustre-MDT0000 complete
[ 5770.852383] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5770.898895] LustreError: 120817:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5770.914119] LustreError: 120817:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message
[ 5771.469585] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5771.549905] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5772.056035] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5772.275728] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5772.369196] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:594 to 0x240000400:609)
[ 5772.370067] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:589 to 0x280000400:609)
[ 5772.463103] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 5777.059440] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5786.138860] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5788.177717] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5793.484053] Lustre: DEBUG MARKER: (dd_pid=114531, time=0, timeout=600)
[ 5847.074577] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5853.241968] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 5856.511209] LustreError: 123010:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5857.900154] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5859.704147] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5861.864622] Lustre: Failing over lustre-MDT0000
[ 5861.987643] LustreError: lustre-MDT0000-lwp-OST0000: operation quota_acquire to node 0@lo failed: rc = -107
[ 5861.991835] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5862.001626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5862.185726] Lustre: server umount lustre-MDT0000 complete
[ 5880.287103] Lustre: 3319:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779992204/real 1779992204]  req@ffff9b307cfc6a00 x1866451033215232/t0(0) o400->MGC192.168.202.128@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1779992220 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5880.325832] Lustre: 3319:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 5880.334923] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5881.375305] Lustre: 3320:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779992204/real 1779992204]  req@ffff9b307cfc5c00 x1866451033215360/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779992220 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5885.558328] Lustre: 3320:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779992209/real 1779992209]  req@ffff9b2f45000700 x1866451033216256/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779992225 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5890.527190] Lustre: 3319:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779992214/real 1779992214]  req@ffff9b307fee9180 x1866451033216768/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779992230 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5890.591591] LustreError: 3316:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight  req@ffff9b307fee8380 x1866451033218176/t0(0) o250->MGC192.168.202.128@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5891.137088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5891.203901] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5892.145457] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 5892.153860] Lustre: Skipped 1 previous similar message
[ 5895.372862] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5900.057643] LustreError: 123623:0:(tgt_handler.c:533:tgt_filter_recovery_request()) @@@ not permitted during recovery  req@ffff9b2f424db800 x1866451033230208/t0(0) o601->lustre-MDT0000-lwp-OST0000_UUID@0@lo:475/0 lens 336/0 e 0 to 0 dl 1779992250 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'qsd_reint_0.lus.0' uid:0 gid:0 projid:4294967295
[ 5900.072768] LustreError: lustre-MDT0000-lwp-OST0000: operation quota_acquire to node 0@lo failed: rc = -11
[ 5904.017030] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5904.224569] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5904.292111] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:589 to 0x280000400:641)
[ 5904.306454] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:611 to 0x240000400:641)
[ 5909.104282] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5911.597401] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5920.692405] Lustre: DEBUG MARKER: (dd_pid=116979, time=2, timeout=600)
[ 5992.565381] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 14:18:49 (1779992329)
[ 6013.737427] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 6015.618892] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 6022.472327] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 6024.097657] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 6058.971358] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 14:19:56 (1779992396)
[ 6084.839182] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 6101.196718] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 6103.325907] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 6105.183202] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 6107.368190] Lustre: DEBUG MARKER: Set quota for 1 times
[ 6111.139434] Lustre: DEBUG MARKER: Set quota for 2 times
[ 6115.229600] Lustre: DEBUG MARKER: Set quota for 3 times
[ 6119.128788] Lustre: DEBUG MARKER: Set quota for 4 times
[ 6122.984920] Lustre: DEBUG MARKER: Set quota for 5 times
[ 6127.852386] Lustre: DEBUG MARKER: Set quota for 6 times
[ 6132.279195] Lustre: DEBUG MARKER: Set quota for 7 times
[ 6136.049879] Lustre: DEBUG MARKER: Set quota for 8 times
[ 6192.727438] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 14:22:10 (1779992530)
[ 6206.946478] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6206.977649] Lustre: Skipped 1 previous similar message
[ 6207.004721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6207.010710] Lustre: Skipped 1 previous similar message
[ 6208.781787] Lustre: server umount lustre-MDT0000 complete
[ 6212.854412] LustreError: 24495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779992552 with bad export cookie 16919876015759251069
[ 6212.863422] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6212.935821] Lustre: server umount lustre-OST0000 complete
[ 6218.265720] Lustre: server umount lustre-OST0001 complete
[ 6239.093367] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local
[ 6247.696832] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6251.344705] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6254.688689] Lustre: 132493:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6260.600516] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 6267.919338] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6272.870977] LustreError: 132814:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6272.907939] LustreError: 132814:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message
[ 6272.930048] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:643 to 0x240000400:673)
[ 6275.650114] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6280.722147] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:644 to 0x280000400:673)
[ 6282.643542] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6290.431422] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6294.045952] Lustre: 134278:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6313.297724] Lustre: server umount lustre-MDT0000 complete
[ 6317.515259] LustreError: 132057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779992657 with bad export cookie 16919876015759258321
[ 6317.523707] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6327.928344] Lustre: server umount lustre-OST0000 complete
[ 6342.967355] Lustre: server umount lustre-OST0001 complete
[ 6360.417258] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local
[ 6371.133088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6376.490809] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6379.728058] Lustre: 136885:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6392.267239] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6395.936884] LustreError: 137202:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6395.992620] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:643 to 0x240000400:705)
[ 6400.451659] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6400.461211] Lustre: Skipped 1 previous similar message
[ 6405.617045] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:644 to 0x280000400:705)
[ 6406.810630] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6414.535111] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6418.267423] Lustre: 138641:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6428.793397] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 14:26:06 (1779992766)
[ 6430.576761] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 6432.400653] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 14:26:09 (1779992769)
[ 6493.752691] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 14:27:11 (1779992831)
[ 6543.173513] Lustre: DEBUG MARKER: Write...
[ 6546.153327] Lustre: DEBUG MARKER: Write out of block quota ...
[ 6603.049860] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 14:29:00 (1779992940)
[ 6610.687955] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 14:29:08 (1779992948)
[ 6621.295557] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 14:29:18 (1779992958)
[ 6631.812439] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 14:29:29 (1779992969)
[ 6642.172266] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 14:29:39 (1779992979)
[ 6710.046259] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 6871.324604] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 7061.260178] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 14:36:38 (1779993398)
[ 7126.993881] Lustre: DEBUG MARKER: Restart...
[ 7135.736227] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 7135.766937] Lustre: Skipped 1 previous similar message
[ 7135.785479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 7135.794907] Lustre: Skipped 1 previous similar message
[ 7136.225635] LustreError: 3318:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED  req@ffff9b304ff64a80 x1866451033608320/t0(0) o601->lustre-MDT0000-lwp-MDT0000@0@lo:23/10 lens 336/336 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 7136.250199] LustreError: 3318:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8  qsd:lustre-MDT0000 qtype:usr lqe: ffff9b30761cb200 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 2660 qunit:0 qtune:0 edquot:0 default:no revoke:0
[ 7138.283153] Lustre: server umount lustre-MDT0000 complete
[ 7142.560754] LustreError: 139391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779993482 with bad export cookie 16919876015759259854
[ 7142.586884] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 7142.702345] Lustre: server umount lustre-OST0000 complete
[ 7147.420660] Lustre: server umount lustre-OST0001 complete
[ 7164.201536] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local
[ 7174.337391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 7178.996930] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7182.115807] Lustre: 157084:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 7187.560178] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 7194.485768] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7198.818863] LustreError: 157403:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 7198.838147] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:714 to 0x240000400:737)
[ 7201.820892] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 7206.923592] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:714 to 0x280000400:737)
[ 7209.036324] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7217.652261] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 7221.211414] Lustre: 158862:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 7292.214218] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 14:40:30 (1779993630)
[ 7368.839543] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 14:41:46 (1779993706)
[ 9022.170845] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 15:09:19 (1779995359)
[ 9040.336936] Lustre: server umount lustre-MDT0000 complete
[ 9046.160555] LustreError: 156651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779995385 with bad export cookie 16919876015759266350
[ 9046.191516] LustreError: 156651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 9046.202849] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 9055.897310] Lustre: server umount lustre-OST0000 complete
[ 9059.167418] Lustre: 3319:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779995382/real 1779995382]  req@ffff9b3058b4bb80 x1866451036584576/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779995398 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 9059.197993] Lustre: 3319:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 9059.208576] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 9059.218022] Lustre: Skipped 1 previous similar message
[ 9060.742449] LustreError: 3317:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8  qsd:lustre-OST0001 qtype:usr lqe: ffff9b2f42f59980 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 2384 qunit:0 qtune:0 edquot:0 default:no revoke:0
[ 9061.068150] Lustre: server umount lustre-OST0001 complete
[ 9078.317481] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local
[ 9089.061607] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 9095.638030] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9099.023145] Lustre: 167181:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 9104.001801] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 9110.631435] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9115.238174] LustreError: 167495:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9115.278857] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5739 to 0x240000400:5761)
[ 9118.752588] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 9123.837642] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5739 to 0x280000400:5761)
[ 9125.891243] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9133.236169] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 9138.537262] Lustre: 168969:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 9175.975316] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 15:11:53 (1779995513)
[ 9222.192794] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 15:12:39 (1779995559)
[ 9269.224379] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 15:13:26 (1779995606)
[ 9271.506976] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs
[ 9273.613814] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 15:13:31 (1779995611)
[ 9275.481281] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs
[ 9277.888087] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 15:13:35 (1779995615)
[ 9375.031117] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 15:15:11 (1779995711)
[ 9439.091661] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 15:16:15 (1779995775)
[ 9562.013707] Lustre: DEBUG MARKER: sanity-quota test_48: @@@@@@ FAIL: QMT: mds1 qid 60000 is not deleted