[ 5190.284728] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5196.280192] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 5199.472589] LustreError: 123237:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5200.843282] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5202.942965] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5205.144396] Lustre: Failing over lustre-MDT0000
[ 5205.348841] Lustre: server umount lustre-MDT0000 complete
[ 5222.111384] Lustre: 3305:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786315711/real 1786315711]  req@ffff9cf209502d80 x1873082438276736/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786315727 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5222.125353] Lustre: 3305:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 5222.128393] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5222.218403] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5222.291680] LustreError: 123882:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5222.670719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5222.773775] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5222.875139] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5223.239764] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5223.309069] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:511 to 0x240000400:545)
[ 5223.315864] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:507 to 0x280000400:545)
[ 5223.717785] Lustre: 3306:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786315711/real 1786315711]  req@ffff9cf209501c00 x1873082438276864/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786315727 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5223.754793] Lustre: 3306:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5227.706545] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5228.022704] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[ 5229.087558] Lustre: 3306:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786315717/real 1786315717]  req@ffff9cf2031bc700 x1873082438277248/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786315733 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5229.120730] Lustre: 3306:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5233.615199] Lustre: 3304:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786315722/real 1786315722]  req@ffff9cf2031bce00 x1873082438277504/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786315738 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5235.972283] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5237.381327] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5245.703473] Lustre: DEBUG MARKER: (dd_pid=115800, time=2, timeout=600)
[ 5292.556573] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5298.463453] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 5301.829896] LustreError: 126237:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5302.969178] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5304.573428] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5306.624026] Lustre: Failing over lustre-MDT0000
[ 5306.931105] Lustre: server umount lustre-MDT0000 complete
[ 5323.231113] Lustre: 3305:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786315812/real 1786315812]  req@ffff9cf337db6680 x1873082438314240/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1786315828 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 5323.279038] Lustre: 3305:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5323.284495] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5323.303516] Lustre: Skipped 1 previous similar message
[ 5323.789260] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5324.195255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5324.247423] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5325.233196] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5325.487711] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5325.539569] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:547 to 0x240000400:577)
[ 5325.542763] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:507 to 0x280000400:577)
[ 5328.161063] LustreError: 3302:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-lwp-OST0001: namespace resource [0x200000006:0x20000:0x0].0x0 (ffff9cf303ede300) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5328.184091] LustreError: 3302:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 5 previous similar messages
[ 5328.198535] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 5328.210138] Lustre: Skipped 1 previous similar message
[ 5328.949587] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5337.077349] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5338.864047] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5339.615607] Lustre: 3305:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786315828/real 1786315828]  req@ffff9cf30afbc000 x1873082438315776/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786315844 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5339.657090] Lustre: 3305:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
[ 5351.207151] Lustre: DEBUG MARKER: (dd_pid=118322, time=7, timeout=600)
[ 5409.005388] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 18:51:52 (1786315912)
[ 5429.545249] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 5431.593570] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 5437.307409] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 5438.831498] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 5473.601246] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 18:52:57 (1786315977)
[ 5502.397817] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 5518.992734] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 5520.431238] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 5522.256439] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 5524.640731] Lustre: DEBUG MARKER: Set quota for 1 times
[ 5528.761389] Lustre: DEBUG MARKER: Set quota for 2 times
[ 5533.176325] Lustre: DEBUG MARKER: Set quota for 3 times
[ 5537.351860] Lustre: DEBUG MARKER: Set quota for 4 times
[ 5541.788667] Lustre: DEBUG MARKER: Set quota for 5 times
[ 5546.496993] Lustre: DEBUG MARKER: Set quota for 6 times
[ 5550.208934] Lustre: DEBUG MARKER: Set quota for 7 times
[ 5613.128991] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 18:55:16 (1786316116)
[ 5625.316131] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5625.323077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5625.358930] Lustre: Skipped 2 previous similar messages
[ 5625.371304] Lustre: Skipped 1 previous similar message
[ 5627.419338] Lustre: server umount lustre-MDT0000 complete
[ 5631.793336] LustreError: 5811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786316136 with bad export cookie 15571983744741060884
[ 5631.794544] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5631.818635] LustreError: 5811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 5631.882454] Lustre: server umount lustre-OST0000 complete
[ 5636.081883] Lustre: server umount lustre-OST0001 complete
[ 5652.211277] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local
[ 5662.688721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5666.968441] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5669.211731] Lustre: 136028:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 5675.112303] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 5681.255771] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5687.333172] LustreError: 136391:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5687.362157] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:579 to 0x240000400:609)
[ 5688.778532] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 5693.955608] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:580 to 0x280000400:609)
[ 5696.193827] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5703.061908] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 5706.306719] Lustre: 137935:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 5724.648229] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5724.649791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5724.662556] Lustre: Skipped 1 previous similar message
[ 5724.702554] Lustre: Skipped 1 previous similar message
[ 5725.074670] Lustre: server umount lustre-MDT0000 complete
[ 5729.302420] LustreError: 135577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786316234 with bad export cookie 15571983744741067212
[ 5729.310442] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5729.450705] Lustre: server umount lustre-OST0000 complete
[ 5733.533216] Lustre: server umount lustre-OST0001 complete
[ 5751.710862] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local
[ 5763.832649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5767.528388] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5770.347434] Lustre: 140644:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 5784.517432] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5789.162484] LustreError: 141014:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5789.243568] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:579 to 0x240000400:641)
[ 5795.179076] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 5795.182373] Lustre: Skipped 1 previous similar message
[ 5800.451270] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:580 to 0x280000400:641)
[ 5802.584500] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5812.251264] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 5815.510107] Lustre: 142557:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 5825.292876] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 18:58:49 (1786316329)
[ 5826.770963] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 5828.328780] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 18:58:52 (1786316332)
[ 5886.126869] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 18:59:49 (1786316389)
[ 5929.465194] Lustre: DEBUG MARKER: Write...
[ 5932.516352] Lustre: DEBUG MARKER: Write out of block quota ...
[ 5988.051983] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 19:01:32 (1786316492)
[ 5996.406475] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 19:01:39 (1786316499)
[ 6006.820594] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 19:01:50 (1786316510)
[ 6015.042587] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 19:01:58 (1786316518)
[ 6025.106176] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 19:02:08 (1786316528)
[ 6088.427111] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 6238.537746] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 6431.955495] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 19:08:55 (1786316935)
[ 6490.479482] Lustre: DEBUG MARKER: Restart...
[ 6494.143439] Lustre: server umount lustre-MDT0000 complete
[ 6497.900673] LustreError: 140199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786317002 with bad export cookie 15571983744741068745
[ 6497.906842] LustreError: 140199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 6497.912021] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6507.541726] Lustre: server umount lustre-OST0000 complete
[ 6520.175072] Lustre: server umount lustre-OST0001 complete
[ 6533.258812] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local
[ 6541.947732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6545.844516] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6548.659355] Lustre: 161411:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6554.264144] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 6560.914403] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6566.505091] LustreError: 161790:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6566.532501] LustreError: 161790:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message
[ 6566.547251] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:650 to 0x240000400:673)
[ 6568.479712] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6573.555540] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:650 to 0x280000400:673)
[ 6574.337089] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6581.331792] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6584.376499] Lustre: 163326:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6645.558322] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 19:12:29 (1786317149)
[ 6716.804944] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 19:13:40 (1786317220)
[ 8170.504653] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 19:37:53 (1786318673)
[ 8184.104163] Lustre: server umount lustre-MDT0000 complete
[ 8187.586607] LustreError: 160966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786318692 with bad export cookie 15571983744741075241
[ 8187.598736] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 8197.149150] Lustre: server umount lustre-OST0000 complete
[ 8211.737698] Lustre: server umount lustre-OST0001 complete
[ 8227.439822] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local
[ 8238.082946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 8243.294184] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8245.752854] Lustre: 171897:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 8251.704744] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 8257.946408] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8263.076432] LustreError: 172266:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 8263.103086] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5675 to 0x240000400:5697)
[ 8266.120382] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 8271.361263] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5675 to 0x280000400:5697)
[ 8272.506669] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8279.090910] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 8282.361962] Lustre: 173799:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 8318.087367] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 19:40:21 (1786318821)
[ 8364.200833] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 19:41:08 (1786318868)
[ 8410.293969] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 19:41:53 (1786318913)
[ 8412.316528] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs
[ 8413.611157] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 19:41:57 (1786318917)
[ 8415.265414] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs
[ 8416.912482] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 19:42:00 (1786318920)
[ 8503.682942] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 19:43:27 (1786319007)
[ 8553.424983] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 19:44:17 (1786319057)
[ 8642.490443] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 19:45:45 (1786319145)
[ 8654.785864] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8654.788367] Lustre: Skipped 1 previous similar message
[ 8655.794636] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8655.804529] Lustre: Skipped 33 previous similar messages
[ 8657.794182] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8657.797291] Lustre: Skipped 108 previous similar messages
[ 8661.800660] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8661.802987] Lustre: Skipped 232 previous similar messages
[ 8669.806528] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8669.809667] Lustre: Skipped 441 previous similar messages
[ 8685.820159] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8685.825144] Lustre: Skipped 963 previous similar messages
[ 8717.824875] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8717.828237] Lustre: Skipped 1935 previous similar messages
[ 8781.831940] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8781.837051] Lustre: Skipped 3857 previous similar messages
[ 9043.940692] Lustre: *** cfs_fail_loc=a08, val=0***
[ 9043.951357] Lustre: Skipped 423 previous similar messages
[ 9043.966132] Lustre: *** cfs_fail_loc=a08, val=0***
[ 9193.463558] Lustre: *** cfs_fail_loc=a08, val=0***
[ 9193.500165] Lustre: Skipped 26 previous similar messages
[ 9223.140448] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 9223.158052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 9224.057265] Lustre: server umount lustre-MDT0000 complete
[ 9227.678683] LustreError: 177391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786319732 with bad export cookie 15571983744742846500
[ 9227.696861] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 9228.005199] Lustre: server umount lustre-OST0000 complete
[ 9231.996611] Lustre: server umount lustre-OST0001 complete
[ 9238.948885] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid
[ 9246.376397] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local
[ 9255.610668]  vdc: vdc1 vdc9
[ 9265.174110]  vde: vde1 vde9
[ 9265.196814]  vde: vde1 vde9
[ 9265.226930]  vde: vde1 vde9
[ 9275.882696]  vdf: vdf1 vdf9
[ 9291.046429] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local
[ 9300.061199] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000
[ 9300.497744] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[ 9300.551770] Lustre: lustre-MDT0000: new disk, initializing
[ 9300.844773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 9300.887678] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[ 9304.970586] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9309.389598] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params
[ 9314.732884] Lustre: lustre-OST0000: new disk, initializing
[ 9314.736719] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[ 9314.741483] Lustre: Skipped 1 previous similar message
[ 9314.836614] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 9316.014316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost
[ 9316.028729] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost]
[ 9316.308404] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400
[ 9320.506354] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9329.383316] Lustre: lustre-OST0001: new disk, initializing
[ 9329.386674] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[ 9329.567318] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 9330.689219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost
[ 9330.698203] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost]
[ 9330.909916] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400
[ 9336.430656] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9346.547738] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 9350.105708] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params
[ 9387.016404] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 19:58:10 (1786319890)
[ 9393.411348] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 19:58:17 (1786319897)
[ 9442.859325] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 19:59:06 (1786319946)
[ 9505.636955] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 20:00:09 (1786320009)
[ 9535.625311] Lustre: DEBUG MARKER: rename directory return 255
[ 9583.230585] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 20:01:26 (1786320086)
[ 9619.066233] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 20:02:02 (1786320122)
[ 9663.070887] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 20:02:46 (1786320166)
[ 9788.199986] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 20:04:51 (1786320291)
[ 9828.399085] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 20:05:31 (1786320331)
[ 9880.111643] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 20:06:23 (1786320383)
[10059.345215] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 20:09:23 (1786320563)
[10062.080471] Lustre: DEBUG MARKER: SKIP: sanity-quota test_59 ldiskfs only test
[10063.630539] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 20:09:27 (1786320567)
[10125.265418] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[10127.430348] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 20:10:30 (1786320630)
[10161.692926] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[10163.401410] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 20:11:06 (1786320666)
[10217.468402] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[10219.356676] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 20:12:02 (1786320722)
[10267.761426] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 20:12:51 (1786320771)
[10269.543810] Lustre: DEBUG MARKER: SKIP: sanity-quota test_67 ZFS grants some block space together with inode
[10271.144894] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 20:12:54 (1786320774)
[10299.260960] LustreError: 212074:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[10344.248216] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 20:14:08 (1786320848)
[10365.641286] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[10367.107059] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[10504.615847] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 20:16:48 (1786321008)
[10557.894729] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 20:17:41 (1786321061)
[10592.300678] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 20:18:15 (1786321095)
[10593.596965] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71a ZFS grants some block space together with inode
[10595.982369] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 20:18:19 (1786321099)
[10597.516218] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71b ZFS grants some block space together with inode
[10599.572299] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 20:18:22 (1786321102)
[10615.406366] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB)
[10632.207589] Lustre: DEBUG MARKER: Write...
[10635.465332] Lustre: DEBUG MARKER: Write out of block quota ...
[10695.316410] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 20:19:58 (1786321198)
[10722.829648] Lustre: DEBUG MARKER: set to use default quota
[10724.530394] Lustre: DEBUG MARKER: set default quota
[10726.468914] Lustre: DEBUG MARKER: get default quota
[10732.295347] Lustre: DEBUG MARKER: Test not out of quota
[10735.726640] Lustre: DEBUG MARKER: Test out of quota
[10746.197973] Lustre: DEBUG MARKER: Increase default quota
[10772.312895] Lustre: DEBUG MARKER: Set quota to override default quota
[10772.356603] LustreError: 214514:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45063 time:1786926077 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[10782.064932] Lustre: DEBUG MARKER: Set to use default quota again
[10800.615526] Lustre: DEBUG MARKER: Cleanup
[10870.001658] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 20:22:53 (1786321373)
[10895.096367] Lustre: DEBUG MARKER: set default quota for qpool1
[10896.195710] Lustre: DEBUG MARKER: Write from user that hasn't lqe
[10945.095123] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 20:24:08 (1786321448)
[11021.374340] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 20:25:25 (1786321525)
[11089.989853] Lustre: DEBUG MARKER: Write...
[11093.270595] Lustre: DEBUG MARKER: Write out of block quota ...
[11207.313957] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 20:28:30 (1786321710)
[11251.459978] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 20:29:14 (1786321754)
[11258.647878] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 20:29:22 (1786321762)
[11260.615703] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78A need >= 2.13.57 and ldiskfs for fallocate
[11262.084667] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 20:29:25 (1786321765)
[11263.790211] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78a need >= 2.13.57 and ldiskfs for fallocate
[11265.642989] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 20:29:29 (1786321769)
[11286.531351] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 20:29:50 (1786321790)
[11288.704111] Lustre: DEBUG MARKER: SKIP: sanity-quota test_80 ZFS grants some block space together with inode
[11290.807367] Lustre: DEBUG MARKER: == sanity-quota test 81: Race qmt_start_pool_recalc with qmt_pool_free ========================================================== 20:29:53 (1786321793)
[11307.024709] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[11313.629728] LustreError: 237181:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 sleeping for 10000ms
[11318.240417] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[11318.241792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[11318.249607] Lustre: Skipped 2 previous similar messages
[11318.252821] Lustre: Skipped 1 previous similar message
[11323.360440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[11323.368563] Lustre: Skipped 3 previous similar messages
[11323.657532] LustreError: 237181:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 awake
[11323.991356] Lustre: server umount lustre-MDT0000 complete
[11330.756967] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[11331.169936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[11331.263346] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:27 to 0x240000400:65)
[11331.269769] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:27 to 0x280000400:65)
[11336.509834] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[11348.962143] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[11348.972344] Lustre: Skipped 1 previous similar message
[11379.492567] Lustre: DEBUG MARKER: == sanity-quota test 82: verify more than 8 qids for single operation ========================================================== 20:31:23 (1786321883)
[11413.652504] Lustre: DEBUG MARKER: == sanity-quota test 83: Setting default quota shouldn't affect grace time ========================================================== 20:31:57 (1786321917)
[11445.881222] Lustre: DEBUG MARKER: == sanity-quota test 84: Reset quota should fix the insane granted quota ========================================================== 20:32:29 (1786321949)
[11492.249043] Lustre: *** cfs_fail_loc=a08, val=0***
[11492.251823] Lustre: Skipped 28 previous similar messages
[11492.256045] Lustre: *** cfs_fail_loc=a08, val=0***
[11492.258636] Lustre: Skipped 4 previous similar messages
[11597.979788] Lustre: DEBUG MARKER: == sanity-quota test 85: do not hung at write with the least_qunit ========================================================== 20:35:01 (1786322101)
[11680.306518] Lustre: DEBUG MARKER: == sanity-quota test 86: Pre-acquired quota should be released if quota is over limit ========================================================== 20:36:24 (1786322184)
[11756.658268] LustreError: 237772:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1786927061 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[11909.724990] LustreError: 237771:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1786927214 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[12070.228498] LustreError: 245880:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:1000 enforced:1 hard:2048 soft:2048 granted:65536 time:1786927375 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[12079.174446] Lustre: DEBUG MARKER: sanity-quota test_86: @@@@@@ FAIL: succeed to create files, expect failed