[ 5814.269524] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5820.721317] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 5824.362281] LustreError: 121487:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5825.302440] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5827.401435] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5829.559186] Lustre: Failing over lustre-MDT0000
[ 5830.064544] Lustre: server umount lustre-MDT0000 complete
[ 5847.519093] Lustre: 3316:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782233889/real 1782233889]  req@ffff9857c484fb80 x1868801744603008/t0(0) o400->MGC192.168.202.106@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1782233905 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5847.519365] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5847.564891] Lustre: 3316:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
[ 5847.564966] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5852.639328] Lustre: 3318:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782233894/real 1782233894]  req@ffff9857c7205f80 x1868801744603776/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782233910 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5852.680118] Lustre: 3318:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5857.375211] Lustre: 3316:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782233899/real 1782233899]  req@ffff9857fb81b480 x1868801744604032/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782233915 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5857.424870] Lustre: 3316:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5857.769077] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x3f6918143a4f24e3
[ 5857.793292] Lustre: MGC192.168.202.106@tcp: Connection restored to 0@lo (at 0@lo)
[ 5858.366427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5858.418676] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5859.421774] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5859.591661] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5859.631286] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:550 to 0x280000400:577)
[ 5859.634351] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:553 to 0x240000400:577)
[ 5862.367485] Lustre: 3315:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782233904/real 1782233904]  req@ffff985708d40000 x1868801744604416/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782233920 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5862.400445] Lustre: 3315:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5862.639565] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5872.551472] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5874.094380] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5891.334458] Lustre: DEBUG MARKER: (dd_pid=114561, time=10, timeout=600)
[ 5941.085585] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5946.987745] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 5950.265648] LustreError: 124431:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5951.174592] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5952.643543] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5954.635428] Lustre: Failing over lustre-MDT0000
[ 5954.999465] Lustre: server umount lustre-MDT0000 complete
[ 5970.916158] Lustre: 3316:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782234012/real 1782234012]  req@ffff9857fb818000 x1868801744645504/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1782234028 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 5970.953492] Lustre: 3316:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5970.966128] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5970.991534] Lustre: Skipped 1 previous similar message
[ 5973.317936] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5974.150525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5974.327928] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5975.093880] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 5975.110571] Lustre: Skipped 2 previous similar messages
[ 5980.489855] LustreError: 125035:0:(tgt_handler.c:534:tgt_filter_recovery_request()) @@@ not permitted during recovery  req@ffff985709fbc000 x1868801744659200/t0(0) o601->lustre-MDT0000-lwp-OST0000_UUID@0@lo:679/0 lens 336/0 e 0 to 0 dl 1782234049 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'qsd_reint_0.lus.0' uid:0 gid:0 projid:4294967295
[ 5980.558210] LustreError: lustre-MDT0000-lwp-OST0000: operation quota_acquire to node 0@lo failed: rc = -11
[ 5980.826623] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5982.229435] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5982.459120] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5982.523908] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:579 to 0x240000400:609)
[ 5982.524304] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:550 to 0x280000400:609)
[ 5991.519471] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5993.611494] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6000.070812] Lustre: DEBUG MARKER: (dd_pid=117045, time=0, timeout=600)
[ 6066.174866] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 13:02:02 (1782234122)
[ 6091.693451] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 6093.531944] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 6100.799295] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 6103.061027] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 6138.913399] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 13:03:15 (1782234195)
[ 6165.401733] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 6182.686914] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 6184.533791] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 6186.074451] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 6188.301681] Lustre: DEBUG MARKER: Set quota for 1 times
[ 6192.472599] Lustre: DEBUG MARKER: Set quota for 2 times
[ 6196.549649] Lustre: DEBUG MARKER: Set quota for 3 times
[ 6200.411422] Lustre: DEBUG MARKER: Set quota for 4 times
[ 6203.975470] Lustre: DEBUG MARKER: Set quota for 5 times
[ 6207.760438] Lustre: DEBUG MARKER: Set quota for 6 times
[ 6211.604275] Lustre: DEBUG MARKER: Set quota for 7 times
[ 6215.686266] Lustre: DEBUG MARKER: Set quota for 8 times
[ 6271.936820] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 13:05:28 (1782234328)
[ 6285.817171] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6285.828723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6285.841770] Lustre: Skipped 2 previous similar messages
[ 6285.854714] Lustre: Skipped 1 previous similar message
[ 6286.200142] Lustre: server umount lustre-MDT0000 complete
[ 6290.879407] LustreError: 20498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782234348 with bad export cookie 4569209772100758814
[ 6290.880446] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6290.895445] LustreError: 20498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message
[ 6290.979451] Lustre: server umount lustre-OST0000 complete
[ 6295.187338] Lustre: server umount lustre-OST0001 complete
[ 6308.873713] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local
[ 6318.058790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6323.067375] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6326.197818] Lustre: 134100:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6331.533714] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 6338.622821] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6342.752890] LustreError: 134414:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6342.791045] LustreError: 134414:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message
[ 6342.792070] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:611 to 0x240000400:641)
[ 6347.103286] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6352.423523] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:612 to 0x280000400:641)
[ 6355.532942] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6364.335965] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6368.106690] Lustre: 135907:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6386.757084] Lustre: server umount lustre-MDT0000 complete
[ 6391.150341] LustreError: 133646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782234449 with bad export cookie 4569209772100765877
[ 6391.161932] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6401.618437] Lustre: server umount lustre-OST0000 complete
[ 6404.584312] Lustre: 3315:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782234446/real 1782234446]  req@ffff9856d3a21f80 x1868801744812928/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782234462 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6404.608713] Lustre: 3315:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
[ 6404.621458] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6406.950366] Lustre: server umount lustre-OST0001 complete
[ 6422.474675] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local
[ 6432.752890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6437.660809] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6440.645415] Lustre: 138517:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6452.293986] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6457.442282] LustreError: 138833:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6457.489807] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:611 to 0x240000400:673)
[ 6461.338812] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6461.349138] Lustre: Skipped 1 previous similar message
[ 6466.593242] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:612 to 0x280000400:673)
[ 6471.031334] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6480.760739] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6485.888341] Lustre: 140332:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6498.879837] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 13:09:14 (1782234554)
[ 6501.236593] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 6503.182287] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 13:09:19 (1782234559)
[ 6561.577419] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 13:10:18 (1782234618)
[ 6609.034044] Lustre: DEBUG MARKER: Write...
[ 6611.599778] Lustre: DEBUG MARKER: Write out of block quota ...
[ 6668.960361] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 13:12:05 (1782234725)
[ 6675.314429] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 13:12:12 (1782234732)
[ 6685.769454] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 13:12:22 (1782234742)
[ 6694.395757] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 13:12:31 (1782234751)
[ 6702.977960] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 13:12:39 (1782234759)
[ 6767.355799] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 6930.124136] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 7120.268034] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 13:19:36 (1782235176)
[ 7184.752625] Lustre: DEBUG MARKER: Restart...
[ 7192.426911] Lustre: server umount lustre-MDT0000 complete
[ 7198.513054] LustreError: 142516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782235256 with bad export cookie 4569209772100767410
[ 7198.517097] LustreError: 3316:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8  qsd:lustre-OST0000 qtype:usr lqe: ffff9857f532db00 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 2378 qunit:0 qtune:0 edquot:0 default:no revoke:0
[ 7198.543958] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 7209.142648] Lustre: server umount lustre-OST0000 complete
[ 7209.954258] Lustre: 3317:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782235251/real 1782235251]  req@ffff9857fc853b80 x1868801745039360/t0(0) o601->lustre-MDT0000-lwp-OST0001@0@lo:23/10 lens 336/336 e 0 to 1 dl 1782235267 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 7209.991457] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 7215.013411] Lustre: server umount lustre-OST0001 complete
[ 7233.051027] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local
[ 7242.355862] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 7246.807840] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7249.484826] Lustre: 158870:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 7255.004910] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 7262.236636] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7267.169601] LustreError: 159188:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 7267.204747] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:683 to 0x240000400:705)
[ 7272.417916] Lustre: lustre-OST0001: Not available for connect from 0@lo (not set up)
[ 7272.880075] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 7278.096412] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:681 to 0x280000400:705)
[ 7280.663784] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7289.480949] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 7293.139207] Lustre: 160687:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 7361.761934] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 13:23:38 (1782235418)
[ 7440.568678] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 13:24:56 (1782235496)
[ 9311.950815] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 13:56:08 (1782237368)
[ 9331.171717] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 9331.188943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 9331.196109] Lustre: Skipped 1 previous similar message
[ 9331.806719] Lustre: server umount lustre-MDT0000 complete
[ 9336.727237] LustreError: 158424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782237394 with bad export cookie 4569209772100773906
[ 9336.734767] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 9336.739382] LustreError: 158424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 9336.817909] Lustre: server umount lustre-OST0000 complete
[ 9341.127803] Lustre: server umount lustre-OST0001 complete
[ 9359.225088] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local
[ 9369.003246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 9373.839460] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9377.267467] Lustre: 168800:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 9383.489345] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 9389.722473] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9393.701892] LustreError: 169119:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9393.747144] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5708 to 0x240000400:5729)
[ 9397.053207] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 9402.366976] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5706 to 0x280000400:5729)
[ 9403.923471] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9413.268199] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 9417.797617] Lustre: 170627:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 9457.010193] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 13:58:33 (1782237513)
[ 9500.164243] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 13:59:16 (1782237556)
[ 9548.287325] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 14:00:04 (1782237604)
[ 9550.192503] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs
[ 9551.878274] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 14:00:08 (1782237608)
[ 9553.610455] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs
[ 9555.517474] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 14:00:12 (1782237612)
[ 9652.356255] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 14:01:48 (1782237708)
[ 9709.633653] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 14:02:45 (1782237765)
[ 9835.907217] Lustre: DEBUG MARKER: sanity-quota test_48: @@@@@@ FAIL: QMT: mds1 qid 60000 is not deleted
[ 9882.123670] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 14:05:38 (1782237938)
[ 9892.860694] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9892.863434] Lustre: Skipped 2 previous similar messages
[ 9894.864098] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9894.866608] Lustre: Skipped 117 previous similar messages
[ 9898.869864] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9898.872342] Lustre: Skipped 209 previous similar messages
[ 9906.873152] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9906.885045] Lustre: Skipped 471 previous similar messages
[ 9922.887428] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9922.889257] Lustre: Skipped 903 previous similar messages
[ 9954.899408] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9954.903258] Lustre: Skipped 1687 previous similar messages
[10018.907775] Lustre: *** cfs_fail_loc=a09, val=0***
[10018.912563] Lustre: Skipped 3775 previous similar messages
[10345.318460] Lustre: *** cfs_fail_loc=a08, val=0***
[10345.327613] Lustre: Skipped 833 previous similar messages
[10345.328657] Lustre: *** cfs_fail_loc=a08, val=0***
[10345.350623] Lustre: Skipped 10 previous similar messages
[10549.217530] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[10549.219810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[10549.240444] Lustre: Skipped 2 previous similar messages
[10554.339256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[10554.353118] Lustre: Skipped 2 previous similar messages
[10555.520549] Lustre: server umount lustre-MDT0000 complete
[10560.973247] LustreError: 168354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782238618 with bad export cookie 4569209772102553229
[10560.990127] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[10561.303487] Lustre: server umount lustre-OST0000 complete
[10566.776784] Lustre: server umount lustre-OST0001 complete
[10575.472750] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid
[10584.926713] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local
[10594.597445]  vdc: vdc1 vdc9
[10594.612398]  vdc: vdc1 vdc9
[10594.651484]  vdc: vdc1 vdc9
[10604.983413]  vde: vde1 vde9
[10604.994476]  vde: vde1 vde9
[10615.567942]  vdf: vdf1 vdf9
[10633.730982] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local
[10642.044368] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000
[10642.323601] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[10642.379041] Lustre: lustre-MDT0000: new disk, initializing
[10642.685528] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[10642.760810] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[10647.083867] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10652.110741] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params
[10658.094841] Lustre: lustre-OST0000: new disk, initializing
[10658.100298] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[10658.103749] Lustre: Skipped 1 previous similar message
[10658.183446] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[10659.332179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost
[10659.352787] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost]
[10659.735718] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400
[10664.722355] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10673.820514] Lustre: lustre-OST0001: new disk, initializing
[10673.824456] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[10673.904490] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[10675.008812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost
[10675.032709] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost]
[10675.266742] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400
[10680.314711] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10689.889786] Lustre: DEBUG MARKER: Using TIMEOUT=20
[10693.699194] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params
[10736.520538] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 14:19:53 (1782238793)
[10744.451509] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 14:20:00 (1782238800)
[10794.111539] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 14:20:50 (1782238850)
[10861.036820] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 14:21:57 (1782238917)
[10890.903546] Lustre: DEBUG MARKER: rename directory return 255
[10939.047942] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 14:23:14 (1782238994)
[10975.821599] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 14:23:51 (1782239031)
[11023.889700] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 14:24:39 (1782239079)
[11150.277318] LustreError: 187464:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-0x0 id:60001 enforced:1 hard:51200 soft:0 granted:51200 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[11194.145309] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 14:27:30 (1782239250)
[11231.439487] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 14:28:08 (1782239288)
[11276.666309] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 14:28:53 (1782239333)
[11455.119041] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 14:31:51 (1782239511)
[11456.530871] Lustre: DEBUG MARKER: SKIP: sanity-quota test_59 ldiskfs only test
[11458.786966] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 14:31:55 (1782239515)
[11521.565886] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[11523.434168] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 14:32:59 (1782239579)
[11556.743585] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[11558.381941] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 14:33:35 (1782239615)
[11612.632829] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[11614.213361] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 14:34:31 (1782239671)
[11663.949705] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 14:35:20 (1782239720)
[11666.063908] Lustre: DEBUG MARKER: SKIP: sanity-quota test_67 ZFS grants some block space together with inode
[11668.240798] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 14:35:24 (1782239724)
[11695.905237] LustreError: 210402:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[11741.354341] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 14:36:37 (1782239797)
[11769.692823] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[11771.422218] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[11928.178889] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 14:39:44 (1782239984)
[11982.462189] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 14:40:39 (1782240039)
[12016.219657] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 14:41:12 (1782240072)
[12017.708995] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71a ZFS grants some block space together with inode
[12019.323465] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 14:41:16 (1782240076)
[12020.970333] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71b ZFS grants some block space together with inode
[12023.039884] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 14:41:19 (1782240079)
[12038.601344] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB)
[12053.827528] Lustre: DEBUG MARKER: Write...
[12055.946948] Lustre: DEBUG MARKER: Write out of block quota ...
[12110.694711] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 14:42:47 (1782240167)
[12138.772552] Lustre: DEBUG MARKER: set to use default quota
[12140.190438] Lustre: DEBUG MARKER: set default quota
[12142.068716] Lustre: DEBUG MARKER: get default quota
[12148.407613] Lustre: DEBUG MARKER: Test not out of quota
[12152.510935] Lustre: DEBUG MARKER: Test out of quota
[12168.731164] Lustre: DEBUG MARKER: Increase default quota
[12200.214625] Lustre: DEBUG MARKER: Set quota to override default quota
[12200.271895] LustreError: 187463:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45063 time:1782845058 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[12213.278510] Lustre: DEBUG MARKER: Set to use default quota again
[12235.892994] Lustre: DEBUG MARKER: Cleanup
[12310.338793] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 14:46:06 (1782240366)
[12338.911264] Lustre: DEBUG MARKER: set default quota for qpool1
[12340.857568] Lustre: DEBUG MARKER: Write from user that hasn't lqe
[12391.009687] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 14:47:27 (1782240447)
[12480.776982] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 14:48:57 (1782240537)
[12558.842731] Lustre: DEBUG MARKER: Write...
[12562.592703] Lustre: DEBUG MARKER: Write out of block quota ...
[12695.025775] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 14:52:31 (1782240751)
[12747.418617] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 14:53:23 (1782240803)
[12754.685338] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 14:53:31 (1782240811)
[12756.871663] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78A need >= 2.13.57 and ldiskfs for fallocate
[12759.045597] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 14:53:35 (1782240815)
[12760.241255] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78a need >= 2.13.57 and ldiskfs for fallocate
[12762.287727] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 14:53:38 (1782240818)
[12783.010748] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 14:53:59 (1782240839)
[12784.337066] Lustre: DEBUG MARKER: SKIP: sanity-quota test_80 ZFS grants some block space together with inode
[12786.023237] Lustre: DEBUG MARKER: == sanity-quota test 81: Race qmt_start_pool_recalc with qmt_pool_free ========================================================== 14:54:02 (1782240842)
[12802.336991] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[12809.049120] LustreError: 235487:0:(qmt_pool.c:1413:qmt_pool_recalc()) cfs_fail_timeout id a07 sleeping for 10000ms
[12809.058715] LustreError: 235487:0:(qmt_pool.c:1413:qmt_pool_recalc()) Skipped 5 previous similar messages
[12813.279739] LustreError: 3316:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED  req@ffff9857fb5af100 x1868801753813632/t0(0) o601->lustre-MDT0000-lwp-MDT0000@0@lo:23/10 lens 336/336 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[12813.305988] LustreError: 3316:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8  qsd:lustre-MDT0000 qtype:prj lqe: ffff985702eb6a80 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 374 qunit:0 qtune:0 edquot:0 default:no revoke:0
[12813.349591] LustreError: 3316:0:(qsd_handler.c:298:qsd_req_completion()) Skipped 2 previous similar messages
[12814.887329] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.6@tcp (stopping)
[12815.330378] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[12815.370665] Lustre: Skipped 1 previous similar message
[12819.071127] LustreError: 235487:0:(qmt_pool.c:1413:qmt_pool_recalc()) cfs_fail_timeout id a07 awake
[12819.343258] Lustre: server umount lustre-MDT0000 complete
[12827.317653] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[12827.816780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[12827.938973] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:27 to 0x280000400:65)
[12827.943102] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:27 to 0x240000400:65)
[12832.361091] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[12835.317468] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[12835.338572] Lustre: Skipped 1 previous similar message
[12868.498370] Lustre: DEBUG MARKER: == sanity-quota test 82: verify more than 8 qids for single operation ========================================================== 14:55:25 (1782240925)
[12903.004618] Lustre: DEBUG MARKER: == sanity-quota test 83: Setting default quota shouldn't affect grace time ========================================================== 14:55:59 (1782240959)
[12937.716704] Lustre: DEBUG MARKER: == sanity-quota test 84: Reset quota should fix the insane granted quota ========================================================== 14:56:34 (1782240994)
[12993.401481] Lustre: *** cfs_fail_loc=a08, val=0***
[12993.414539] Lustre: Skipped 7 previous similar messages
[12993.423702] Lustre: *** cfs_fail_loc=a08, val=0***
[12993.429000] Lustre: Skipped 6 previous similar messages
[13114.960924] Lustre: DEBUG MARKER: == sanity-quota test 85: do not hung at write with the least_qunit ========================================================== 14:59:31 (1782241171)
[13206.658224] Lustre: DEBUG MARKER: == sanity-quota test 86: Pre-acquired quota should be released if quota is over limit ========================================================== 15:01:03 (1782241263)
[13282.345253] LustreError: 236031:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1782846140 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[13291.431637] Lustre: DEBUG MARKER: sanity-quota test_86: @@@@@@ FAIL: succeed to create files, expect failed