[ 5216.711391] Lustre: DEBUG MARKER: User quota (limit: 200) [ 5224.194475] Lustre: DEBUG MARKER: Write 100M (buffered) ... [ 5228.632151] LustreError: 121574:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 5230.236257] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5233.400684] Lustre: DEBUG MARKER: Fail mds for 40 seconds [ 5236.623281] Lustre: Failing over lustre-MDT0000 [ 5237.167306] Lustre: server umount lustre-MDT0000 complete [ 5252.576321] Lustre: 3320:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778657353/real 1778657353] req@ffff9c8d8e09c380 x1865052060708224/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1778657369 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5252.581345] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5252.594348] Lustre: 3320:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 5252.595803] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5252.604362] Lustre: Skipped 2 previous similar messages [ 5253.599387] Lustre: 3323:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778657353/real 1778657353] req@ffff9c8eb5160e00 x1865052060708480/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1778657369 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295 [ 5253.641099] Lustre: 3323:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 5257.759185] Lustre: 3321:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778657358/real 1778657358] req@ffff9c8e96aeb480 x1865052060708736/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1778657374 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5257.798131] Lustre: 3321:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 5262.735217] Lustre: 3320:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778657363/real 1778657363] req@ffff9c8d8e09d180 x1865052060709248/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1778657379 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5262.817660] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x122080f57757ecc [ 5262.844531] Lustre: MGC192.168.204.131@tcp: Connection restored to 0@lo (at 0@lo) [ 5262.858949] Lustre: Skipped 1 previous similar message [ 5263.385290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5263.464557] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5267.603373] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 5267.841420] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 5267.889505] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:573 to 0x240000400:609) [ 5267.895203] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:569 to 0x280000400:609) [ 5268.205541] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5276.182248] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5277.918458] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5300.457490] Lustre: DEBUG MARKER: (dd_pid=115385, time=15, timeout=600) [ 5353.871651] Lustre: DEBUG MARKER: User quota (limit: 200) [ 5362.767480] Lustre: DEBUG MARKER: Write 100M (directio) ... [ 5366.442111] LustreError: 124414:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 5367.569782] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5369.240641] Lustre: DEBUG MARKER: Fail mds for 40 seconds [ 5371.232418] Lustre: Failing over lustre-MDT0000 [ 5371.688494] Lustre: server umount lustre-MDT0000 complete [ 5388.259059] Lustre: 3323:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778657488/real 1778657488] req@ffff9c8dc7ea3800 x1865052060753024/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1778657504 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295 [ 5388.311550] Lustre: 3323:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 5388.323271] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5389.602270] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5390.102138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5390.179895] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5390.544022] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 5390.716935] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 5390.772733] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:569 to 0x280000400:641) [ 5390.774671] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:611 to 0x240000400:641) [ 5393.063643] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 5393.071941] Lustre: Skipped 2 previous similar messages [ 5394.404583] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5402.791385] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5404.674601] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5419.452724] Lustre: DEBUG MARKER: (dd_pid=117859, time=8, timeout=600) [ 5487.246175] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 03:33:23 (1778657603) [ 5513.252921] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1: [ 5514.612207] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2). [ 5520.149418] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1: [ 5521.419393] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2). [ 5556.314567] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 03:34:31 (1778657671) [ 5582.916478] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing [ 5599.442792] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr [ 5601.237751] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr [ 5602.908599] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000 [ 5605.059617] Lustre: DEBUG MARKER: Set quota for 1 times [ 5608.725713] Lustre: DEBUG MARKER: Set quota for 2 times [ 5612.280733] Lustre: DEBUG MARKER: Set quota for 3 times [ 5615.734530] Lustre: DEBUG MARKER: Set quota for 4 times [ 5619.511751] Lustre: DEBUG MARKER: Set quota for 5 times [ 5623.240672] Lustre: DEBUG MARKER: Set quota for 6 times [ 5627.360572] Lustre: DEBUG MARKER: Set quota for 7 times [ 5631.778248] Lustre: DEBUG MARKER: Set quota for 8 times [ 5692.324457] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 03:36:47 (1778657807) [ 5705.698641] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5705.706857] Lustre: Skipped 1 previous similar message [ 5705.727687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5706.159077] Lustre: server umount lustre-MDT0000 complete [ 5710.522847] LustreError: 5773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778657826 with bad export cookie 81636605230942106 [ 5710.526650] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5710.535391] LustreError: 5773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5710.655583] Lustre: server umount lustre-OST0000 complete [ 5714.677663] Lustre: server umount lustre-OST0001 complete [ 5729.436344] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 5739.604435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5745.394263] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5748.480796] Lustre: 133944:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5754.250317] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5760.560091] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5764.458302] LustreError: 134265:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5764.505580] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:644 to 0x240000400:673) [ 5764.509745] LustreError: 134265:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 5768.326496] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5773.319480] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:643 to 0x280000400:673) [ 5774.092710] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5782.956770] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5786.965243] Lustre: 135732:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5804.517582] Lustre: server umount lustre-MDT0000 complete [ 5809.239310] LustreError: 133511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778657925 with bad export cookie 81636605230949407 [ 5809.257605] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5818.967217] Lustre: server umount lustre-OST0000 complete [ 5833.786060] Lustre: server umount lustre-OST0001 complete [ 5850.706937] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 5862.100659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5867.192917] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5870.665466] Lustre: 138306:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5878.336458] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5887.137574] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5887.459068] LustreError: 138619:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5887.470781] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:644 to 0x240000400:705) [ 5892.590984] LustreError: 138619:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5901.863975] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:643 to 0x280000400:705) [ 5904.014849] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5912.692915] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5916.454991] Lustre: 140095:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5926.973466] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 03:40:42 (1778658042) [ 5928.543309] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS [ 5930.966739] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 03:40:46 (1778658046) [ 5991.322491] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 03:41:46 (1778658106) [ 6044.430936] Lustre: DEBUG MARKER: Write... [ 6047.444136] Lustre: DEBUG MARKER: Write out of block quota ... [ 6102.239518] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 03:43:38 (1778658218) [ 6108.766050] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 03:43:44 (1778658224) [ 6118.645805] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 03:43:54 (1778658234) [ 6127.573728] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 03:44:03 (1778658243) [ 6135.781681] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 03:44:11 (1778658251) [ 6199.464425] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user [ 6356.088258] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user [ 6544.168877] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 03:50:59 (1778658659) [ 6608.655992] Lustre: DEBUG MARKER: Restart... [ 6616.545507] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6616.547047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6616.559949] Lustre: Skipped 2 previous similar messages [ 6616.566146] Lustre: Skipped 1 previous similar message [ 6619.960996] Lustre: server umount lustre-MDT0000 complete [ 6624.361396] LustreError: 137879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778658740 with bad export cookie 81636605230950940 [ 6624.370730] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6624.373049] LustreError: 137879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6624.458922] Lustre: server umount lustre-OST0000 complete [ 6628.886429] Lustre: server umount lustre-OST0001 complete [ 6644.516679] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 6655.061947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6655.066147] Lustre: Skipped 1 previous similar message [ 6659.733376] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6662.749668] Lustre: 158714:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6668.496691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6675.885873] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6679.714894] LustreError: 159031:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6679.751854] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:714 to 0x240000400:737) [ 6684.653700] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6689.827019] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:714 to 0x280000400:737) [ 6691.709979] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6699.590430] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6703.162887] Lustre: 160496:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6774.678367] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 03:54:50 (1778658890) [ 6851.469347] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 03:56:06 (1778658966) [ 8626.772590] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 04:25:42 (1778660742) [ 8645.603103] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8645.605055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8645.618422] Lustre: Skipped 1 previous similar message [ 8645.629941] Lustre: Skipped 2 previous similar messages [ 8646.524831] Lustre: server umount lustre-MDT0000 complete [ 8650.866239] LustreError: 158285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778660767 with bad export cookie 81636605230957429 [ 8650.879138] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8650.959830] Lustre: server umount lustre-OST0000 complete [ 8656.478242] Lustre: server umount lustre-OST0001 complete [ 8675.741898] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 8687.601101] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8693.099930] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8696.665798] Lustre: 168849:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8703.237783] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8709.777515] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8712.353352] LustreError: 169166:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8712.383261] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5740 to 0x240000400:5761) [ 8717.977507] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8723.442972] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5738 to 0x280000400:5761) [ 8724.794692] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8733.013595] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8737.669609] Lustre: 170634:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8773.432818] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 04:28:08 (1778660888) [ 8821.198158] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 04:28:56 (1778660936) [ 8870.244148] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 04:29:45 (1778660985) [ 8872.527622] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs [ 8874.421822] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 04:29:50 (1778660990) [ 8876.335392] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs [ 8878.591935] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 04:29:53 (1778660993) [ 8971.650855] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 04:31:27 (1778661087) [ 9025.328606] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 04:32:20 (1778661140) [ 9150.406184] Lustre: DEBUG MARKER: sanity-quota test_48: @@@@@@ FAIL: QMT: mds1 qid 60000 is not deleted