[ 6058.833877] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 6065.016547] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 6068.426701] LustreError: 120848:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 6069.470768] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 6071.576961] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 6074.253786] Lustre: Failing over lustre-MDT0000
[ 6074.572493] Lustre: server umount lustre-MDT0000 complete
[ 6091.743858] Lustre: 3322:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786064957/real 1786064957]  req@ffff8b3bee559500 x1872818647084544/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1786064973 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 6091.785947] Lustre: 3322:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 6091.791331] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6092.838604] Lustre: 3322:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786064958/real 1786064958]  req@ffff8b3bee76a300 x1872818647084672/t0(0) o400->MGC192.168.204.139@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786064974 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6092.865448] Lustre: 3322:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 6092.876655] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6097.952149] Lustre: 3319:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786064963/real 1786064963]  req@ffff8b3ac1919c00 x1872818647085312/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786064979 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6103.073518] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x4e9fd959a0f93c33
[ 6103.081820] Lustre: MGC192.168.204.139@tcp: Connection restored to 0@lo (at 0@lo)
[ 6103.612229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6103.711399] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 6104.095131] Lustre: 3322:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786064968/real 1786064968]  req@ffff8b3bee559f80 x1872818647085568/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786064984 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6104.114162] Lustre: 3322:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 6104.514724] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 6104.835694] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 6104.958075] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:496 to 0x240000400:513)
[ 6104.958076] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:491 to 0x280000400:513)
[ 6109.450887] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6120.822643] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6123.138941] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6135.173897] Lustre: DEBUG MARKER: (dd_pid=114313, time=5, timeout=600)
[ 6184.862239] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 6190.845660] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 6194.692411] LustreError: 123730:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 6195.726628] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 6197.666921] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 6199.934534] Lustre: Failing over lustre-MDT0000
[ 6200.351884] Lustre: server umount lustre-MDT0000 complete
[ 6218.283509] Lustre: 3322:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786065083/real 1786065083]  req@ffff8b3ac1919c00 x1872818647128064/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1786065099 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 6218.318230] Lustre: 3322:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 6218.327505] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6218.342729] Lustre: Skipped 1 previous similar message
[ 6218.948762] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6219.439509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6219.552878] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 6223.336774] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 6223.348433] Lustre: Skipped 2 previous similar messages
[ 6225.020863] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6228.404466] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 6228.536745] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 6228.586443] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:491 to 0x280000400:545)
[ 6228.587392] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:515 to 0x240000400:545)
[ 6234.096738] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6235.104480] Lustre: 3321:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786065099/real 1786065099]  req@ffff8b3ac41f6680 x1872818647129472/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786065115 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6235.143283] Lustre: 3321:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
[ 6235.429120] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6245.872525] Lustre: DEBUG MARKER: (dd_pid=116725, time=5, timeout=600)
[ 6313.464325] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 21:13:12 (1786065192)
[ 6338.808894] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 6340.852279] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 6347.499375] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 6349.410253] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 6385.095987] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 21:14:23 (1786065263)
[ 6415.150838] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 6433.646708] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 6435.571539] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 6437.619911] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 6440.127364] Lustre: DEBUG MARKER: Set quota for 1 times
[ 6445.432332] Lustre: DEBUG MARKER: Set quota for 2 times
[ 6450.967212] Lustre: DEBUG MARKER: Set quota for 3 times
[ 6456.658097] Lustre: DEBUG MARKER: Set quota for 4 times
[ 6460.999875] Lustre: DEBUG MARKER: Set quota for 5 times
[ 6465.634033] Lustre: DEBUG MARKER: Set quota for 6 times
[ 6469.945190] Lustre: DEBUG MARKER: Set quota for 7 times
[ 6514.096722] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 21:16:32 (1786065392)
[ 6528.491670] Lustre: server umount lustre-MDT0000 complete
[ 6533.159274] LustreError: 5876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786065414 with bad export cookie 5665485835231380583
[ 6533.168323] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6543.409524] Lustre: server umount lustre-OST0000 complete
[ 6546.399980] Lustre: 3321:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786065411/real 1786065411]  req@ffff8b3b019c9180 x1872818647250560/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786065427 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6546.460889] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6546.484085] Lustre: Skipped 1 previous similar message
[ 6548.068381] Lustre: server umount lustre-OST0001 complete
[ 6563.493681] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local
[ 6575.225255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6580.506902] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6583.493208] Lustre: 133477:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6589.362535] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 6596.072311] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6600.553532] LustreError: 133841:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6600.598700] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:547 to 0x240000400:577)
[ 6605.069903] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6610.426630] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:548 to 0x280000400:577)
[ 6611.822167] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6619.109430] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6622.664705] Lustre: 135394:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6639.009077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6639.017559] Lustre: Skipped 1 previous similar message
[ 6641.196613] Lustre: server umount lustre-MDT0000 complete
[ 6645.388165] LustreError: 133025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786065526 with bad export cookie 5665485835231386806
[ 6645.392097] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6645.398332] LustreError: 133025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message
[ 6645.489063] Lustre: server umount lustre-OST0000 complete
[ 6649.118352] Lustre: server umount lustre-OST0001 complete
[ 6667.365499] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local
[ 6679.476739] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6684.993396] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6688.146852] Lustre: 138053:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6700.935870] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6704.234287] LustreError: 138416:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6704.277661] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:547 to 0x240000400:609)
[ 6709.461283] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6709.464890] Lustre: Skipped 1 previous similar message
[ 6714.865789] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:548 to 0x280000400:609)
[ 6715.359377] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6722.397926] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6726.982488] Lustre: 139962:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6739.135685] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 21:20:17 (1786065617)
[ 6740.915826] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 6742.983533] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 21:20:21 (1786065621)
[ 6799.994419] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 21:21:18 (1786065678)
[ 6848.318704] Lustre: DEBUG MARKER: Write...
[ 6851.045088] Lustre: DEBUG MARKER: Write out of block quota ...
[ 6905.917405] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 21:23:04 (1786065784)
[ 6913.564984] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 21:23:12 (1786065792)
[ 6923.844180] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 21:23:23 (1786065803)
[ 6933.147880] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 21:23:32 (1786065812)
[ 6942.525460] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 21:23:41 (1786065821)
[ 7008.450121] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 7153.798160] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 7339.391788] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 21:30:18 (1786066218)
[ 7406.050018] Lustre: DEBUG MARKER: Restart...
[ 7410.125779] Lustre: server umount lustre-MDT0000 complete
[ 7414.888601] LustreError: 139966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786066296 with bad export cookie 5665485835231388339
[ 7414.910332] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 7425.661205] Lustre: server umount lustre-OST0000 complete
[ 7427.567863] Lustre: 3319:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786066292/real 1786066292]  req@ffff8b3bfb91b100 x1872818647509888/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786066308 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7427.612567] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 7427.634857] Lustre: Skipped 2 previous similar messages
[ 7431.180798] Lustre: server umount lustre-OST0001 complete
[ 7448.980833] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local
[ 7458.248774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 7462.488494] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7464.903029] Lustre: 158715:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 7470.183124] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 7476.353935] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7483.425414] LustreError: 159085:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 7483.458832] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:618 to 0x240000400:641)
[ 7485.583560] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 7490.555238] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:618 to 0x280000400:641)
[ 7491.496636] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7497.920844] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 7500.988524] Lustre: 160630:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 7563.643266] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 21:34:02 (1786066442)
[ 7640.707640] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 21:35:19 (1786066519)
[ 9285.961166] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 22:02:44 (1786068164)
[ 9301.356836] Lustre: server umount lustre-MDT0000 complete
[ 9305.386213] LustreError: 160632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786068186 with bad export cookie 5665485835231394828
[ 9305.397899] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 9314.906126] Lustre: server umount lustre-OST0000 complete
[ 9319.135318] Lustre: 3320:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786068184/real 1786068184]  req@ffff8b3bf63b8e00 x1872818650440704/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786068200 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 9319.178555] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 9319.590654] Lustre: server umount lustre-OST0001 complete
[ 9337.643784] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local
[ 9351.445346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 9356.820529] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9359.873744] Lustre: 169057:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 9367.355105] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 9375.145726] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9376.550553] LustreError: 169431:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9376.746705] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5643 to 0x240000400:5665)
[ 9381.859470] LustreError: 169431:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9385.366708] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 9390.594298] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5643 to 0x280000400:5665)
[ 9393.395876] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9402.084637] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 9405.935910] Lustre: 170992:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 9445.059476] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 22:05:23 (1786068323)
[ 9489.888855] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 22:06:08 (1786068368)
[ 9536.633574] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 22:06:55 (1786068415)
[ 9538.559300] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs
[ 9540.886853] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 22:06:59 (1786068419)
[ 9542.626926] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs
[ 9544.527655] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 22:07:03 (1786068423)
[ 9638.696932] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 22:08:37 (1786068517)
[ 9694.245869] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 22:09:32 (1786068572)
[ 9783.375634] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 22:11:02 (1786068662)
[ 9795.753356] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9795.760343] Lustre: Skipped 3 previous similar messages
[ 9797.841075] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9797.843836] Lustre: Skipped 91 previous similar messages
[ 9801.862357] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9801.866165] Lustre: Skipped 215 previous similar messages
[ 9809.863065] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9809.866140] Lustre: Skipped 417 previous similar messages
[ 9825.864664] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9825.867877] Lustre: Skipped 907 previous similar messages
[ 9857.865808] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9857.867774] Lustre: Skipped 1655 previous similar messages
[ 9921.881223] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9921.884946] Lustre: Skipped 3229 previous similar messages
[10330.915197] Lustre: *** cfs_fail_loc=a08, val=0***
[10330.921803] Lustre: *** cfs_fail_loc=a08, val=0***
[10330.923282] Lustre: Skipped 1483 previous similar messages
[10515.937152] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[10515.938062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[10515.946107] Lustre: Skipped 1 previous similar message
[10515.949621] Lustre: Skipped 1 previous similar message
[10521.060746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[10521.081228] Lustre: Skipped 1 previous similar message
[10522.234144] Lustre: server umount lustre-MDT0000 complete
[10527.127654] LustreError: 174364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786069408 with bad export cookie 5665485835233169566
[10527.145709] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[10527.469209] Lustre: server umount lustre-OST0000 complete
[10532.944771] Lustre: server umount lustre-OST0001 complete
[10542.765144] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_hostid
[10551.602442] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local
[10561.075809]  vdc: vdc1 vdc9
[10570.961037]  vde: vde1 vde9
[10580.969632]  vdf: vdf1 vdf9
[10580.989588]  vdf: vdf1 vdf9
[10581.020858]  vdf: vdf1 vdf9
[10602.669819] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local
[10612.779549] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000
[10613.109559] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[10613.216903] Lustre: lustre-MDT0000: new disk, initializing
[10613.744872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[10613.829754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[10619.915545] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10625.418196] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params
[10632.641384] Lustre: lustre-OST0000: new disk, initializing
[10632.643932] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[10632.647276] Lustre: Skipped 1 previous similar message
[10632.731894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[10634.299406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost
[10634.306827] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost]
[10634.442641] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400
[10639.551793] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10651.388271] Lustre: lustre-OST0001: new disk, initializing
[10651.391615] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[10651.510719] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[10652.761451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost
[10652.766532] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost]
[10652.974659] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400
[10657.645911] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10667.760379] Lustre: DEBUG MARKER: Using TIMEOUT=20
[10671.841791] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params
[10710.669462] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 22:26:29 (1786069589)
[10717.021258] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 22:26:36 (1786069596)
[10767.341031] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 22:27:26 (1786069646)
[10829.768181] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 22:28:28 (1786069708)
[10861.491721] Lustre: DEBUG MARKER: rename directory return 255
[10905.828275] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 22:29:44 (1786069784)
[10940.265581] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 22:30:19 (1786069819)
[10988.885656] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 22:31:07 (1786069867)
[11088.854890] LustreError: 185615:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-0x0 id:60001 enforced:1 hard:51200 soft:0 granted:51200 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[11131.288930] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 22:33:29 (1786070009)
[11151.246979] LustreError: 3319:0:(qsd_reint.c:633:qqi_reint_delayed()) lustre-OST0001: Delaying reintegration for qtype:0 until pending updates are flushed.
[11176.443583] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 22:34:15 (1786070055)
[11222.254766] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 22:35:01 (1786070101)
[11411.719835] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 22:38:10 (1786070290)
[11413.828934] Lustre: DEBUG MARKER: SKIP: sanity-quota test_59 ldiskfs only test
[11416.144474] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 22:38:14 (1786070294)
[11479.613028] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[11481.768797] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 22:39:20 (1786070360)
[11519.552922] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[11521.801634] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 22:40:00 (1786070400)
[11579.124994] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[11582.214930] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 22:41:00 (1786070460)
[11633.979924] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 22:41:52 (1786070512)
[11635.997157] Lustre: DEBUG MARKER: SKIP: sanity-quota test_67 ZFS grants some block space together with inode
[11638.376818] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 22:41:57 (1786070517)
[11666.736223] LustreError: 208718:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[11712.619696] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 22:43:11 (1786070591)
[11740.315527] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[11742.195747] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[11932.495740] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 22:46:51 (1786070811)
[11986.592407] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 22:47:45 (1786070865)
[12025.848690] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 22:48:24 (1786070904)
[12028.220106] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71a ZFS grants some block space together with inode
[12030.914794] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 22:48:29 (1786070909)
[12032.684841] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71b ZFS grants some block space together with inode
[12035.217711] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 22:48:33 (1786070913)
[12052.824551] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB)
[12069.080108] Lustre: DEBUG MARKER: Write...
[12071.744954] Lustre: DEBUG MARKER: Write out of block quota ...
[12129.403180] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 22:50:08 (1786071008)
[12157.452120] Lustre: DEBUG MARKER: set to use default quota
[12159.285212] Lustre: DEBUG MARKER: set default quota
[12161.456724] Lustre: DEBUG MARKER: get default quota
[12167.741381] Lustre: DEBUG MARKER: Test not out of quota
[12172.391986] Lustre: DEBUG MARKER: Test out of quota
[12183.838814] Lustre: DEBUG MARKER: Increase default quota
[12226.046358] Lustre: DEBUG MARKER: Set quota to override default quota
[12226.152735] LustreError: 185611:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45063 time:1786675907 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[12238.290212] Lustre: DEBUG MARKER: Set to use default quota again
[12257.804671] Lustre: DEBUG MARKER: Cleanup
[12332.314416] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 22:53:31 (1786071211)
[12358.566783] Lustre: DEBUG MARKER: set default quota for qpool1
[12360.615735] Lustre: DEBUG MARKER: Write from user that hasn't lqe
[12415.041457] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 22:54:54 (1786071294)
[12508.657181] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 22:56:27 (1786071387)
[12591.948788] Lustre: DEBUG MARKER: Write...
[12595.516797] Lustre: DEBUG MARKER: Write out of block quota ...
[12719.436775] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 22:59:58 (1786071598)
[12767.475408] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 23:00:46 (1786071646)
[12775.114772] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 23:00:53 (1786071653)
[12776.905521] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78A need >= 2.13.57 and ldiskfs for fallocate
[12779.018759] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 23:00:57 (1786071657)
[12781.388590] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78a need >= 2.13.57 and ldiskfs for fallocate
[12782.990879] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 23:01:02 (1786071662)
[12806.468640] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 23:01:24 (1786071684)
[12808.506497] Lustre: DEBUG MARKER: SKIP: sanity-quota test_80 ZFS grants some block space together with inode
[12810.358702] Lustre: DEBUG MARKER: == sanity-quota test 81: Race qmt_start_pool_recalc with qmt_pool_free ========================================================== 23:01:29 (1786071689)
[12830.556202] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[12838.122378] LustreError: 233862:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 sleeping for 10000ms
[12838.128765] LustreError: 233862:0:(qmt_pool.c:1407:qmt_pool_recalc()) Skipped 5 previous similar messages
[12840.931685] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[12840.938946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[12840.959605] Lustre: Skipped 1 previous similar message
[12840.997088] Lustre: Skipped 1 previous similar message
[12841.952150] LustreError: 3322:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED  req@ffff8b3bc4a22a00 x1872818656268416/t0(0) o601->lustre-MDT0000-lwp-MDT0000@0@lo:23/10 lens 336/336 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[12841.980633] LustreError: 3320:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8  qsd:lustre-MDT0000 qtype:grp lqe: ffff8b3aff0a0f00 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 372 qunit:0 qtune:0 edquot:0 default:no revoke:0
[12842.003760] LustreError: 3320:0:(qsd_handler.c:298:qsd_req_completion()) Skipped 3 previous similar messages
[12845.001441] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.39@tcp (stopping)
[12846.052304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[12846.059805] Lustre: Skipped 1 previous similar message
[12848.183185] LustreError: 233862:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 awake
[12848.744720] Lustre: server umount lustre-MDT0000 complete
[12861.558768] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[12862.891741] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[12863.141680] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:27 to 0x280000400:65)
[12863.154322] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:27 to 0x240000400:65)
[12869.656443] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[12871.140607] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[12871.145641] Lustre: Skipped 1 previous similar message
[12907.070350] Lustre: DEBUG MARKER: == sanity-quota test 82: verify more than 8 qids for single operation ========================================================== 23:03:05 (1786071785)
[12941.915901] Lustre: DEBUG MARKER: == sanity-quota test 83: Setting default quota shouldn't affect grace time ========================================================== 23:03:41 (1786071821)
[12982.189650] Lustre: DEBUG MARKER: == sanity-quota test 84: Reset quota should fix the insane granted quota ========================================================== 23:04:20 (1786071860)
[13040.342051] Lustre: *** cfs_fail_loc=a08, val=0***
[13040.353989] Lustre: Skipped 11 previous similar messages
[13040.370036] Lustre: *** cfs_fail_loc=a08, val=0***
[13040.379342] Lustre: Skipped 15 previous similar messages
[13160.683508] Lustre: DEBUG MARKER: == sanity-quota test 85: do not hung at write with the least_qunit ========================================================== 23:07:18 (1786072038)
[13246.496610] Lustre: DEBUG MARKER: == sanity-quota test 86: Pre-acquired quota should be released if quota is over limit ========================================================== 23:08:45 (1786072125)
[13334.272737] LustreError: 234462:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1786677015 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[13506.283259] LustreError: 239390:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1786677187 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[13514.529924] Lustre: DEBUG MARKER: sanity-quota test_86: @@@@@@ FAIL: succeed to create files, expect failed