[ 5814.269524] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5820.721317] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 5824.362281] LustreError: 121487:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5825.302440] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5827.401435] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5829.559186] Lustre: Failing over lustre-MDT0000
[ 5830.064544] Lustre: server umount lustre-MDT0000 complete
[ 5847.519093] Lustre: 3316:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782233889/real 1782233889]  req@ffff9857c484fb80 x1868801744603008/t0(0) o400->MGC192.168.202.106@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1782233905 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5847.519365] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5847.564891] Lustre: 3316:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
[ 5847.564966] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5852.639328] Lustre: 3318:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782233894/real 1782233894]  req@ffff9857c7205f80 x1868801744603776/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782233910 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5852.680118] Lustre: 3318:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5857.375211] Lustre: 3316:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782233899/real 1782233899]  req@ffff9857fb81b480 x1868801744604032/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782233915 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5857.424870] Lustre: 3316:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5857.769077] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x3f6918143a4f24e3
[ 5857.793292] Lustre: MGC192.168.202.106@tcp: Connection restored to 0@lo (at 0@lo)
[ 5858.366427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5858.418676] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5859.421774] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5859.591661] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5859.631286] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:550 to 0x280000400:577)
[ 5859.634351] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:553 to 0x240000400:577)
[ 5862.367485] Lustre: 3315:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782233904/real 1782233904]  req@ffff985708d40000 x1868801744604416/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782233920 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5862.400445] Lustre: 3315:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5862.639565] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5872.551472] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5874.094380] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5891.334458] Lustre: DEBUG MARKER: (dd_pid=114561, time=10, timeout=600)
[ 5941.085585] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5946.987745] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 5950.265648] LustreError: 124431:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5951.174592] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5952.643543] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5954.635428] Lustre: Failing over lustre-MDT0000
[ 5954.999465] Lustre: server umount lustre-MDT0000 complete
[ 5970.916158] Lustre: 3316:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782234012/real 1782234012]  req@ffff9857fb818000 x1868801744645504/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1782234028 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 5970.953492] Lustre: 3316:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5970.966128] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5970.991534] Lustre: Skipped 1 previous similar message
[ 5973.317936] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5974.150525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5974.327928] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5975.093880] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 5975.110571] Lustre: Skipped 2 previous similar messages
[ 5980.489855] LustreError: 125035:0:(tgt_handler.c:534:tgt_filter_recovery_request()) @@@ not permitted during recovery  req@ffff985709fbc000 x1868801744659200/t0(0) o601->lustre-MDT0000-lwp-OST0000_UUID@0@lo:679/0 lens 336/0 e 0 to 0 dl 1782234049 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'qsd_reint_0.lus.0' uid:0 gid:0 projid:4294967295
[ 5980.558210] LustreError: lustre-MDT0000-lwp-OST0000: operation quota_acquire to node 0@lo failed: rc = -11
[ 5980.826623] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5982.229435] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5982.459120] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5982.523908] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:579 to 0x240000400:609)
[ 5982.524304] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:550 to 0x280000400:609)
[ 5991.519471] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5993.611494] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6000.070812] Lustre: DEBUG MARKER: (dd_pid=117045, time=0, timeout=600)
[ 6066.174866] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 13:02:02 (1782234122)
[ 6091.693451] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 6093.531944] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 6100.799295] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 6103.061027] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 6138.913399] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 13:03:15 (1782234195)
[ 6165.401733] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 6182.686914] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 6184.533791] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 6186.074451] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 6188.301681] Lustre: DEBUG MARKER: Set quota for 1 times
[ 6192.472599] Lustre: DEBUG MARKER: Set quota for 2 times
[ 6196.549649] Lustre: DEBUG MARKER: Set quota for 3 times
[ 6200.411422] Lustre: DEBUG MARKER: Set quota for 4 times
[ 6203.975470] Lustre: DEBUG MARKER: Set quota for 5 times
[ 6207.760438] Lustre: DEBUG MARKER: Set quota for 6 times
[ 6211.604275] Lustre: DEBUG MARKER: Set quota for 7 times
[ 6215.686266] Lustre: DEBUG MARKER: Set quota for 8 times
[ 6271.936820] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 13:05:28 (1782234328)
[ 6285.817171] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6285.828723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6285.841770] Lustre: Skipped 2 previous similar messages
[ 6285.854714] Lustre: Skipped 1 previous similar message
[ 6286.200142] Lustre: server umount lustre-MDT0000 complete
[ 6290.879407] LustreError: 20498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782234348 with bad export cookie 4569209772100758814
[ 6290.880446] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6290.895445] LustreError: 20498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message
[ 6290.979451] Lustre: server umount lustre-OST0000 complete
[ 6295.187338] Lustre: server umount lustre-OST0001 complete
[ 6308.873713] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local
[ 6318.058790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6323.067375] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6326.197818] Lustre: 134100:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6331.533714] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 6338.622821] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6342.752890] LustreError: 134414:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6342.791045] LustreError: 134414:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message
[ 6342.792070] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:611 to 0x240000400:641)
[ 6347.103286] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6352.423523] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:612 to 0x280000400:641)
[ 6355.532942] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6364.335965] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6368.106690] Lustre: 135907:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6386.757084] Lustre: server umount lustre-MDT0000 complete
[ 6391.150341] LustreError: 133646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782234449 with bad export cookie 4569209772100765877
[ 6391.161932] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6401.618437] Lustre: server umount lustre-OST0000 complete
[ 6404.584312] Lustre: 3315:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782234446/real 1782234446]  req@ffff9856d3a21f80 x1868801744812928/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782234462 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6404.608713] Lustre: 3315:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
[ 6404.621458] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6406.950366] Lustre: server umount lustre-OST0001 complete
[ 6422.474675] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local
[ 6432.752890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6437.660809] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6440.645415] Lustre: 138517:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6452.293986] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6457.442282] LustreError: 138833:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6457.489807] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:611 to 0x240000400:673)
[ 6461.338812] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6461.349138] Lustre: Skipped 1 previous similar message
[ 6466.593242] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:612 to 0x280000400:673)
[ 6471.031334] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6480.760739] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6485.888341] Lustre: 140332:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6498.879837] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 13:09:14 (1782234554)
[ 6501.236593] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 6503.182287] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 13:09:19 (1782234559)
[ 6561.577419] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 13:10:18 (1782234618)
[ 6609.034044] Lustre: DEBUG MARKER: Write...
[ 6611.599778] Lustre: DEBUG MARKER: Write out of block quota ...
[ 6668.960361] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 13:12:05 (1782234725)
[ 6675.314429] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 13:12:12 (1782234732)
[ 6685.769454] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 13:12:22 (1782234742)
[ 6694.395757] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 13:12:31 (1782234751)
[ 6702.977960] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 13:12:39 (1782234759)
[ 6767.355799] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 6930.124136] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 7120.268034] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 13:19:36 (1782235176)
[ 7184.752625] Lustre: DEBUG MARKER: Restart...
[ 7192.426911] Lustre: server umount lustre-MDT0000 complete
[ 7198.513054] LustreError: 142516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782235256 with bad export cookie 4569209772100767410
[ 7198.517097] LustreError: 3316:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8  qsd:lustre-OST0000 qtype:usr lqe: ffff9857f532db00 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 2378 qunit:0 qtune:0 edquot:0 default:no revoke:0
[ 7198.543958] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 7209.142648] Lustre: server umount lustre-OST0000 complete
[ 7209.954258] Lustre: 3317:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782235251/real 1782235251]  req@ffff9857fc853b80 x1868801745039360/t0(0) o601->lustre-MDT0000-lwp-OST0001@0@lo:23/10 lens 336/336 e 0 to 1 dl 1782235267 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 7209.991457] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 7215.013411] Lustre: server umount lustre-OST0001 complete
[ 7233.051027] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local
[ 7242.355862] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 7246.807840] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7249.484826] Lustre: 158870:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 7255.004910] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 7262.236636] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7267.169601] LustreError: 159188:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 7267.204747] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:683 to 0x240000400:705)
[ 7272.417916] Lustre: lustre-OST0001: Not available for connect from 0@lo (not set up)
[ 7272.880075] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 7278.096412] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:681 to 0x280000400:705)
[ 7280.663784] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7289.480949] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 7293.139207] Lustre: 160687:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 7361.761934] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 13:23:38 (1782235418)
[ 7440.568678] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 13:24:56 (1782235496)
[ 9311.950815] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 13:56:08 (1782237368)
[ 9331.171717] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 9331.188943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 9331.196109] Lustre: Skipped 1 previous similar message
[ 9331.806719] Lustre: server umount lustre-MDT0000 complete
[ 9336.727237] LustreError: 158424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782237394 with bad export cookie 4569209772100773906
[ 9336.734767] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 9336.739382] LustreError: 158424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 9336.817909] Lustre: server umount lustre-OST0000 complete
[ 9341.127803] Lustre: server umount lustre-OST0001 complete
[ 9359.225088] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local
[ 9369.003246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 9373.839460] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9377.267467] Lustre: 168800:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 9383.489345] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 9389.722473] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9393.701892] LustreError: 169119:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9393.747144] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5708 to 0x240000400:5729)
[ 9397.053207] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 9402.366976] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5706 to 0x280000400:5729)
[ 9403.923471] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9413.268199] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 9417.797617] Lustre: 170627:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 9457.010193] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 13:58:33 (1782237513)
[ 9500.164243] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 13:59:16 (1782237556)
[ 9548.287325] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 14:00:04 (1782237604)
[ 9550.192503] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs
[ 9551.878274] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 14:00:08 (1782237608)
[ 9553.610455] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs
[ 9555.517474] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 14:00:12 (1782237612)
[ 9652.356255] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 14:01:48 (1782237708)
[ 9709.633653] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 14:02:45 (1782237765)
[ 9835.907217] Lustre: DEBUG MARKER: sanity-quota test_48: @@@@@@ FAIL: QMT: mds1 qid 60000 is not deleted