[ 5534.065044] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5540.340170] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 5543.685149] LustreError: 121299:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5544.659633] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5546.371355] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5548.592949] Lustre: Failing over lustre-MDT0000
[ 5549.177527] Lustre: server umount lustre-MDT0000 complete
[ 5566.433101] Lustre: 3306:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787853522/real 1787853522]  req@ffff8fa588bf8780 x1874694545484032/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1787853538 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 5566.479080] Lustre: 3306:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 5566.491117] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5567.227476] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5567.704888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5567.764479] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5568.549687] Lustre: 3306:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787853524/real 1787853524]  req@ffff8fa58895a580 x1874694545484416/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787853540 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5568.581213] Lustre: 3306:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5569.117469] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5569.253173] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5569.311051] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:552 to 0x280000400:577)
[ 5569.324458] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:556 to 0x240000400:577)
[ 5571.687273] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 5572.077625] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5574.624943] Lustre: 3308:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787853530/real 1787853530]  req@ffff8fa550a52940 x1874694545484672/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787853546 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5579.936537] Lustre: 3308:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787853535/real 1787853535]  req@ffff8fa545468000 x1874694545485184/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787853551 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5579.963524] Lustre: 3308:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5581.041205] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5582.412040] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5594.108310] Lustre: DEBUG MARKER: (dd_pid=114792, time=6, timeout=600)
[ 5638.585449] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5645.079302] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 5648.244800] LustreError: 124204:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5649.072690] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5650.567583] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5652.413120] Lustre: Failing over lustre-MDT0000
[ 5652.631060] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.40@tcp (stopping)
[ 5652.931711] Lustre: server umount lustre-MDT0000 complete
[ 5669.344222] Lustre: 3305:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787853625/real 1787853625]  req@ffff8fa543b67c00 x1874694545522560/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1787853641 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 5669.349313] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5669.368257] Lustre: 3305:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5669.375910] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5669.398261] Lustre: Skipped 2 previous similar messages
[ 5680.290215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5680.373848] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5685.339911] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5692.538598] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5692.682774] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5692.733897] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:552 to 0x280000400:609)
[ 5692.736243] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:579 to 0x240000400:609)
[ 5694.502125] LustreError: 3304:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-lwp-OST0001: namespace resource [0x200000006:0x20000:0x0].0x0 (ffff8fa67afe1b00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5694.518346] LustreError: 3304:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 5 previous similar messages
[ 5694.523454] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 5694.527462] Lustre: Skipped 1 previous similar message
[ 5698.135833] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5699.597429] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5711.199572] Lustre: DEBUG MARKER: (dd_pid=117230, time=6, timeout=600)
[ 5771.749775] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 14:02:21 (1787853741)
[ 5791.069358] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 5792.377542] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 5797.967710] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 5799.662075] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 5832.970392] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 14:03:22 (1787853802)
[ 5857.386587] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 5875.106821] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 5877.226700] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 5879.178507] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 5881.800282] Lustre: DEBUG MARKER: Set quota for 1 times
[ 5885.665924] Lustre: DEBUG MARKER: Set quota for 2 times
[ 5889.188817] Lustre: DEBUG MARKER: Set quota for 3 times
[ 5892.851922] Lustre: DEBUG MARKER: Set quota for 4 times
[ 5897.209155] Lustre: DEBUG MARKER: Set quota for 5 times
[ 5900.584429] Lustre: DEBUG MARKER: Set quota for 6 times
[ 5904.508939] Lustre: DEBUG MARKER: Set quota for 7 times
[ 5908.170106] Lustre: DEBUG MARKER: Set quota for 8 times
[ 5966.951569] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 14:05:36 (1787853936)
[ 5979.111063] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5979.133401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5981.619810] Lustre: server umount lustre-MDT0000 complete
[ 5985.517605] LustreError: 5822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853957 with bad export cookie 2610407961261400860
[ 5985.526715] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5985.528546] LustreError: 5822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 5985.642046] Lustre: server umount lustre-OST0000 complete
[ 6000.299258] Lustre: server umount lustre-OST0001 complete
[ 6015.432651] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local
[ 6026.409689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6030.953347] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6034.081941] Lustre: 134058:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6041.092922] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 6047.484515] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6051.298197] LustreError: 134436:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6051.328455] LustreError: 134436:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message
[ 6051.355395] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:612 to 0x240000400:641)
[ 6055.770318] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6061.061255] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:611 to 0x280000400:641)
[ 6063.064219] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6071.613993] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6075.384870] Lustre: 135985:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6093.256934] Lustre: server umount lustre-MDT0000 complete
[ 6097.873689] LustreError: 136234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854069 with bad export cookie 2610407961261408007
[ 6097.900098] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6108.250770] Lustre: server umount lustre-OST0000 complete
[ 6112.228461] Lustre: 3305:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787854068/real 1787854068]  req@ffff8fa674238b40 x1874694545687808/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787854084 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6112.267912] Lustre: 3305:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
[ 6112.286238] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6113.320771] Lustre: server umount lustre-OST0001 complete
[ 6134.171653] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local
[ 6147.204390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6152.449168] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6155.572445] Lustre: 138636:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6168.537682] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6171.624628] LustreError: 138992:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6171.652768] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:612 to 0x240000400:673)
[ 6176.738175] LustreError: 139514:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6177.556525] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6177.562807] Lustre: Skipped 1 previous similar message
[ 6182.898337] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:611 to 0x280000400:673)
[ 6184.731054] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6192.819595] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6196.518951] Lustre: 140540:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6208.009215] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 14:09:37 (1787854177)
[ 6209.827342] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 6211.536790] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 14:09:40 (1787854180)
[ 6266.265761] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 14:10:35 (1787854235)
[ 6317.606239] Lustre: DEBUG MARKER: Write...
[ 6320.492141] Lustre: DEBUG MARKER: Write out of block quota ...
[ 6380.677659] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 14:12:29 (1787854349)
[ 6388.465854] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 14:12:37 (1787854357)
[ 6398.407598] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 14:12:47 (1787854367)
[ 6406.688750] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 14:12:55 (1787854375)
[ 6415.305174] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 14:13:04 (1787854384)
[ 6480.847550] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 6637.353284] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 6816.512846] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 14:19:45 (1787854785)
[ 6878.490504] Lustre: DEBUG MARKER: Restart...
[ 6882.276183] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6882.284405] LustreError: 138206:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6882.296294] LustreError: 138206:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message
[ 6882.426249] Lustre: server umount lustre-MDT0000 complete
[ 6885.563797] LustreError: 142660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854857 with bad export cookie 2610407961261409540
[ 6885.568091] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6885.575686] LustreError: 142660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 6885.672879] Lustre: server umount lustre-OST0000 complete
[ 6889.023473] Lustre: server umount lustre-OST0001 complete
[ 6902.214619] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local
[ 6911.519729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6915.743879] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6918.476407] Lustre: 159138:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6924.110661] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 6929.243402] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6936.294516] LustreError: 159506:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6936.312850] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:683 to 0x240000400:705)
[ 6936.699719] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6941.700272] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:681 to 0x280000400:705)
[ 6942.620382] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6950.627441] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6954.559641] Lustre: 161051:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 7019.294799] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 14:23:08 (1787854988)
[ 7097.242867] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 14:24:26 (1787855066)
[ 8861.932504] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 14:53:51 (1787856831)
[ 8876.046073] Lustre: server umount lustre-MDT0000 complete
[ 8879.996242] LustreError: 158692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856851 with bad export cookie 2610407961261416043
[ 8880.007899] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 8889.312442] LustreError: 3307:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED  req@ffff8fa5854cb840 x1874694548892544/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 8889.337295] LustreError: 3307:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8  qsd:lustre-OST0000 qtype:usr lqe: ffff8fa543ab8600 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 2384 qunit:0 qtune:0 edquot:0 default:no revoke:0
[ 8889.463680] Lustre: server umount lustre-OST0000 complete
[ 8892.449829] Lustre: 3307:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787856848/real 1787856848]  req@ffff8fa5854c9680 x1874694548891776/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787856864 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 8892.500253] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 8892.519808] Lustre: Skipped 1 previous similar message
[ 8894.261634] Lustre: server umount lustre-OST0001 complete
[ 8910.370246] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local
[ 8920.150682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 8924.456250] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8927.184833] Lustre: 169754:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 8933.975272] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 8939.961410] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8944.161730] LustreError: 170127:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 8944.206892] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5708 to 0x240000400:5729)
[ 8947.688805] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 8952.827898] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5706 to 0x280000400:5729)
[ 8953.406293] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8960.940782] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 8964.578675] Lustre: 171684:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 8999.514361] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 14:56:08 (1787856968)
[ 9044.236923] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 14:56:53 (1787857013)
[ 9090.189486] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 14:57:39 (1787857059)
[ 9091.760862] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs
[ 9093.135884] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 14:57:42 (1787857062)
[ 9094.910738] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs
[ 9097.064492] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 14:57:46 (1787857066)
[ 9188.612107] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 14:59:17 (1787857157)
[ 9238.008658] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 15:00:07 (1787857207)
[ 9322.729157] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 15:01:32 (1787857292)
[ 9333.453189] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9333.460110] Lustre: Skipped 2 previous similar messages
[ 9334.462570] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9334.465207] Lustre: Skipped 45 previous similar messages
[ 9336.489637] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9336.497838] Lustre: Skipped 95 previous similar messages
[ 9340.501418] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9340.509153] Lustre: Skipped 175 previous similar messages
[ 9348.526542] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9348.536537] Lustre: Skipped 339 previous similar messages
[ 9364.531092] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9364.535642] Lustre: Skipped 716 previous similar messages
[ 9396.541134] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9396.543124] Lustre: Skipped 1474 previous similar messages
[ 9460.559527] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9460.561560] Lustre: Skipped 2785 previous similar messages
[ 9872.745224] Lustre: *** cfs_fail_loc=a08, val=0***
[ 9872.754982] Lustre: Skipped 2369 previous similar messages
[ 9872.763388] Lustre: *** cfs_fail_loc=a08, val=0***
[ 9872.775056] Lustre: Skipped 7 previous similar messages
[10110.947168] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[10110.948456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[10110.952391] Lustre: Skipped 1 previous similar message
[10110.965870] Lustre: Skipped 1 previous similar message
[10116.066315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[10116.073955] Lustre: Skipped 1 previous similar message
[10117.610852] Lustre: server umount lustre-MDT0000 complete
[10121.653719] LustreError: 169305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858093 with bad export cookie 2610407961263194848
[10121.674609] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[10122.151591] Lustre: server umount lustre-OST0000 complete
[10126.284978] Lustre: server umount lustre-OST0001 complete
[10132.953992] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_hostid
[10141.497789] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local
[10150.676874]  vdc: vdc1 vdc9
[10160.096770]  vde: vde1 vde9
[10169.982332]  vdf: vdf1 vdf9
[10187.153229] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local
[10196.649471] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000
[10196.888164] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[10196.947765] Lustre: lustre-MDT0000: new disk, initializing
[10197.265380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[10197.316178] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[10201.868724] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10206.699697] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params
[10212.821240] Lustre: lustre-OST0000: new disk, initializing
[10212.823913] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[10212.827273] Lustre: Skipped 1 previous similar message
[10212.897134] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[10214.030152] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost
[10214.047918] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost]
[10214.178938] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400
[10219.033523] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10229.163258] Lustre: lustre-OST0001: new disk, initializing
[10229.167867] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[10229.253404] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[10230.489994] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost
[10230.498519] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost]
[10230.691814] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400
[10235.705881] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10245.676033] Lustre: DEBUG MARKER: Using TIMEOUT=20
[10249.807613] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params
[10287.464577] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 15:17:36 (1787858256)
[10293.967852] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 15:17:43 (1787858263)
[10342.797720] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 15:18:32 (1787858312)
[10403.561264] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 15:19:32 (1787858372)
[10433.068251] Lustre: DEBUG MARKER: rename directory return 255
[10485.155754] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 15:20:53 (1787858453)
[10521.788867] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 15:21:30 (1787858490)
[10567.396840] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 15:22:16 (1787858536)
[10718.376397] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 15:24:47 (1787858687)
[10757.896995] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 15:25:26 (1787858726)
[10801.921724] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 15:26:11 (1787858771)
[10986.337821] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 15:29:15 (1787858955)
[10987.823351] Lustre: DEBUG MARKER: SKIP: sanity-quota test_59 ldiskfs only test
[10989.451654] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 15:29:18 (1787858958)
[11045.245360] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[11046.781295] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 15:30:16 (1787859016)
[11078.794841] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[11081.188906] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 15:30:49 (1787859049)
[11134.062526] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[11135.808276] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 15:31:45 (1787859105)
[11184.579670] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 15:32:33 (1787859153)
[11186.468563] Lustre: DEBUG MARKER: SKIP: sanity-quota test_67 ZFS grants some block space together with inode
[11188.319829] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 15:32:37 (1787859157)
[11217.816701] LustreError: 209841:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[11266.501844] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 15:33:55 (1787859235)
[11295.307156] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[11297.082626] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[11480.866382] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 15:37:30 (1787859450)
[11534.747455] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 15:38:23 (1787859503)
[11572.165141] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 15:39:01 (1787859541)
[11574.241556] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71a ZFS grants some block space together with inode
[11576.682439] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 15:39:05 (1787859545)
[11578.389213] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71b ZFS grants some block space together with inode
[11580.848691] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 15:39:09 (1787859549)
[11599.524996] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB)
[11618.667105] Lustre: DEBUG MARKER: Write...
[11622.440767] Lustre: DEBUG MARKER: Write out of block quota ...
[11686.984365] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 15:40:55 (1787859655)
[11714.539510] Lustre: DEBUG MARKER: set to use default quota
[11716.639836] Lustre: DEBUG MARKER: set default quota
[11718.753551] Lustre: DEBUG MARKER: get default quota
[11725.761258] Lustre: DEBUG MARKER: Test not out of quota
[11730.473253] Lustre: DEBUG MARKER: Test out of quota
[11743.072847] Lustre: DEBUG MARKER: Increase default quota
[11772.790743] Lustre: DEBUG MARKER: Set quota to override default quota
[11772.851291] LustreError: 186742:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45063 time:1788464544 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[11785.952590] Lustre: DEBUG MARKER: Set to use default quota again
[11810.093265] Lustre: DEBUG MARKER: Cleanup
[11882.939448] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 15:44:12 (1787859852)
[11907.542619] Lustre: DEBUG MARKER: set default quota for qpool1
[11909.911528] Lustre: DEBUG MARKER: Write from user that hasn't lqe
[11961.424269] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 15:45:30 (1787859930)
[12049.112658] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 15:46:58 (1787860018)
[12124.852229] Lustre: DEBUG MARKER: Write...
[12128.151696] Lustre: DEBUG MARKER: Write out of block quota ...
[12251.350872] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 15:50:20 (1787860220)
[12300.048413] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 15:51:09 (1787860269)
[12307.348751] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 15:51:16 (1787860276)
[12308.996914] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78A need >= 2.13.57 and ldiskfs for fallocate
[12311.067767] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 15:51:20 (1787860280)
[12312.869960] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78a need >= 2.13.57 and ldiskfs for fallocate
[12315.174159] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 15:51:24 (1787860284)
[12334.073946] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 15:51:43 (1787860303)
[12335.530780] Lustre: DEBUG MARKER: SKIP: sanity-quota test_80 ZFS grants some block space together with inode
[12337.039937] Lustre: DEBUG MARKER: == sanity-quota test 81: Race qmt_start_pool_recalc with qmt_pool_free ========================================================== 15:51:46 (1787860306)
[12353.269372] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[12359.156302] LustreError: 234911:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 sleeping for 10000ms
[12359.171279] LustreError: 234911:0:(qmt_pool.c:1407:qmt_pool_recalc()) Skipped 2 previous similar messages
[12366.309205] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[12366.310338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[12366.339647] Lustre: Skipped 1 previous similar message
[12366.360959] Lustre: Skipped 1 previous similar message
[12367.003231] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.40@tcp (stopping)
[12369.224465] LustreError: 234911:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 awake
[12369.513602] Lustre: server umount lustre-MDT0000 complete
[12377.216202] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[12377.819640] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[12377.926957] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:27 to 0x240000400:65)
[12377.927643] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:27 to 0x280000400:65)
[12381.928481] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[12386.282310] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[12386.290949] Lustre: Skipped 1 previous similar message
[12418.785327] Lustre: DEBUG MARKER: == sanity-quota test 82: verify more than 8 qids for single operation ========================================================== 15:53:07 (1787860387)
[12453.586321] Lustre: DEBUG MARKER: == sanity-quota test 83: Setting default quota shouldn't affect grace time ========================================================== 15:53:42 (1787860422)
[12486.195488] Lustre: DEBUG MARKER: == sanity-quota test 84: Reset quota should fix the insane granted quota ========================================================== 15:54:15 (1787860455)
[12539.984756] Lustre: *** cfs_fail_loc=a08, val=0***
[12539.995347] Lustre: Skipped 8 previous similar messages
[12540.003031] Lustre: *** cfs_fail_loc=a08, val=0***
[12540.007975] Lustre: Skipped 8 previous similar messages
[12660.861495] Lustre: DEBUG MARKER: == sanity-quota test 85: do not hung at write with the least_qunit ========================================================== 15:57:10 (1787860630)
[12749.391198] Lustre: DEBUG MARKER: == sanity-quota test 86: Pre-acquired quota should be released if quota is over limit ========================================================== 15:58:38 (1787860718)
[12831.137415] LustreError: 236008:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1788465602 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[13009.536913] LustreError: 236009:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1788465781 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[13018.199696] Lustre: DEBUG MARKER: sanity-quota test_86: @@@@@@ FAIL: succeed to create files, expect failed