[ 4223.014345] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 4227.519216] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 4230.334769] LustreError: 97387:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 4231.236908] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4232.845827] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 4234.741675] Lustre: Failing over lustre-MDT0000
[ 4235.042926] LustreError: 97580:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup
[ 4235.045683] LustreError: 97580:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message
[ 4235.243062] Lustre: server umount lustre-MDT0000 complete
[ 4251.369987] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4251.622251] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 4251.630065] Lustre: Skipped 1 previous similar message
[ 4251.827070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4251.885914] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 4252.704057] Lustre: 3334:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761249618/real 1761249618]  req@ffff99fa53d71180 x1846799699775488/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761249634 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4252.746732] Lustre: 3334:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 4253.776103] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 4253.937141] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 4253.993170] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:809 to 0x240000400:833)
[ 4253.999880] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:807 to 0x280000400:833)
[ 4255.487531] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4256.739567] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 4257.874633] LustreError: 6578:0:(qsd_reint.c:618:qqi_reint_delayed()) lustre-OST0000: Delaying reintegration for qtype:0 until pending updates are flushed.
[ 4257.887317] LustreError: 6578:0:(qsd_reint.c:618:qqi_reint_delayed()) Skipped 6 previous similar messages
[ 4257.951186] Lustre: 3337:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761249623/real 1761249623]  req@ffff99fa53d71c00 x1846799699775872/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761249639 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4262.216290] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4262.879661] Lustre: 3334:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761249628/real 1761249628]  req@ffff99fb74e75500 x1846799699776128/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761249644 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4262.903227] Lustre: 3334:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 4264.406618] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4269.447099] Lustre: DEBUG MARKER: (dd_pid=93930, time=0, timeout=600)
[ 4314.273549] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 4318.802727] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 4321.525697] LustreError: 99936:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 4322.467379] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4323.878813] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 4325.770068] Lustre: Failing over lustre-MDT0000
[ 4326.014231] LustreError: lustre-MDT0000-lwp-OST0000: operation quota_acquire to node 0@lo failed: rc = -107
[ 4326.017479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 4328.131901] LustreError: 100131:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup
[ 4328.135311] LustreError: 100131:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 4328.354819] Lustre: server umount lustre-MDT0000 complete
[ 4343.903465] Lustre: 3337:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761249709/real 1761249709]  req@ffff99fb75280380 x1846799699809792/t0(0) o400->MGC192.168.203.145@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761249725 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4343.927697] Lustre: 3337:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 4343.931688] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4354.147797] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x65348bdbb358327c
[ 4354.812773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4354.918485] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 4355.813190] Lustre: 3337:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761249720/real 1761249720]  req@ffff99fb7d718a80 x1846799699811328/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761249736 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4355.867432] Lustre: 3337:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 4356.167804] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 4356.328280] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 4356.378189] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:835 to 0x240000400:865)
[ 4356.378351] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:807 to 0x280000400:865)
[ 4358.549420] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4363.355119] LustreError: 33432:0:(qsd_reint.c:618:qqi_reint_delayed()) lustre-OST0000: Delaying reintegration for qtype:0 until pending updates are flushed.
[ 4363.368978] LustreError: 33432:0:(qsd_reint.c:618:qqi_reint_delayed()) Skipped 1 previous similar message
[ 4365.053692] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4366.544800] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4369.379328] LustreError: 3333:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-lwp-OST0001: namespace resource [0x200000006:0x20000:0x0].0x0 (ffff99fb48cf7000) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4369.399360] LustreError: 3333:0:(ldlm_resource.c:981:ldlm_resource_complain()) Skipped 5 previous similar messages
[ 4382.110466] Lustre: DEBUG MARKER: (dd_pid=96220, time=9, timeout=600)
[ 4433.661446] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 16:03:33 (1761249813)
[ 4445.109244] LustreError: 102560:0:(qsd_reint.c:618:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:1 until pending updates are flushed.
[ 4451.365090] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 4452.999619] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 4457.791352] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 4459.230948] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 4490.300551] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 16:04:29 (1761249869)
[ 4508.360558] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 4521.294097] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 4522.622647] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 4523.834490] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 4525.177864] Lustre: DEBUG MARKER: Set quota for 1 times
[ 4528.257949] Lustre: DEBUG MARKER: Set quota for 2 times
[ 4531.179713] Lustre: DEBUG MARKER: Set quota for 3 times
[ 4534.222581] Lustre: DEBUG MARKER: Set quota for 4 times
[ 4537.218213] Lustre: DEBUG MARKER: Set quota for 5 times
[ 4540.035394] Lustre: DEBUG MARKER: Set quota for 6 times
[ 4543.005254] Lustre: DEBUG MARKER: Set quota for 7 times
[ 4546.166483] Lustre: DEBUG MARKER: Set quota for 8 times
[ 4549.294560] Lustre: DEBUG MARKER: Set quota for 9 times
[ 4552.449593] Lustre: DEBUG MARKER: Set quota for 10 times
[ 4592.820388] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 16:06:12 (1761249972)
[ 4595.306088] LustreError: 106115:0:(qsd_reint.c:618:qqi_reint_delayed()) lustre-OST0001: Delaying reintegration for qtype:2 until pending updates are flushed.
[ 4604.896294] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 4604.916511] Lustre: Skipped 2 previous similar messages
[ 4604.920207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 4604.932995] Lustre: Skipped 1 previous similar message
[ 4609.668319] LustreError: 106517:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup
[ 4609.672906] LustreError: 106517:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 4609.828631] Lustre: server umount lustre-MDT0000 complete
[ 4613.580857] LustreError: 18559:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761249995 with bad export cookie 7292607472342676092
[ 4613.590583] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4613.598469] LustreError: 18559:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 4613.724888] Lustre: server umount lustre-OST0000 complete
[ 4617.542543] Lustre: server umount lustre-OST0001 complete
[ 4629.722565] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local
[ 4637.923031] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4641.265362] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4643.934129] Lustre: 108388:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 4648.931307] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 4653.526595] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4659.729299] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 4662.767164] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:868 to 0x280000400:897)
[ 4662.771753] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:867 to 0x240000400:897)
[ 4664.128095] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4671.722255] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 4675.242269] Lustre: 110169:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 4693.440384] LustreError: 110870:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup
[ 4693.442351] LustreError: 110870:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages
[ 4693.473035] LustreError: 107966:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 4693.485928] LustreError: 107966:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message
[ 4693.567415] Lustre: server umount lustre-MDT0000 complete
[ 4696.921171] LustreError: 110418:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761250078 with bad export cookie 7292607472342684030
[ 4696.933051] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4696.946584] LustreError: 110418:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message
[ 4697.050769] Lustre: server umount lustre-OST0000 complete
[ 4700.609961] Lustre: server umount lustre-OST0001 complete
[ 4713.401261] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local
[ 4720.780063] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4723.982373] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4726.288585] Lustre: 112738:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 4731.682052] LustreError: 113066:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 4731.702853] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:867 to 0x240000400:929)
[ 4735.796431] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4736.994688] LustreError: 113066:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 4741.587104] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 4741.589961] Lustre: Skipped 1 previous similar message
[ 4743.491889] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:868 to 0x280000400:929)
[ 4746.332833] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4752.988566] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 4755.865353] Lustre: 114526:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 4768.799596] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 16:09:08 (1761250148)
[ 4770.015871] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 4771.262542] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 16:09:11 (1761250151)
[ 4815.662781] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 16:09:55 (1761250195)
[ 4855.957906] Lustre: DEBUG MARKER: Write...
[ 4858.132247] Lustre: DEBUG MARKER: Write out of block quota ...
[ 4904.650275] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 16:11:23 (1761250283)
[ 4910.670351] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 16:11:30 (1761250290)
[ 4918.188957] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 16:11:38 (1761250298)
[ 4925.162660] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 16:11:44 (1761250304)
[ 4931.531894] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 16:11:51 (1761250311)
[ 4983.790695] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 5093.411021] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 5234.359410] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 16:16:54 (1761250614)
[ 5275.391704] Lustre: DEBUG MARKER: Restart...
[ 5279.201831] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5279.204519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5279.212226] Lustre: Skipped 4 previous similar messages
[ 5279.221963] Lustre: Skipped 1 previous similar message
[ 5279.908689] LustreError: 127414:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup
[ 5279.911420] LustreError: 127414:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages
[ 5280.056376] Lustre: server umount lustre-MDT0000 complete
[ 5282.979135] LustreError: 112309:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761250664 with bad export cookie 7292607472342685563
[ 5282.983048] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5283.003517] LustreError: 112309:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 5283.097467] Lustre: server umount lustre-OST0000 complete
[ 5286.474300] Lustre: server umount lustre-OST0001 complete
[ 5296.430654] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local
[ 5303.780389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5307.147470] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5309.594976] Lustre: 129267:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 5319.925798] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5328.945048] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:937 to 0x280000400:961)
[ 5328.963092] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:939 to 0x240000400:961)
[ 5332.267955] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5338.173081] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 5340.732768] Lustre: 131045:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 5393.734540] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 16:19:33 (1761250773)
[ 5448.123209] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 16:20:27 (1761250827)
[ 6781.593798] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 16:42:41 (1761252161)
[ 6796.696852] LustreError: 135864:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup
[ 6796.699465] LustreError: 135864:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages
[ 6796.817280] Lustre: server umount lustre-MDT0000 complete
[ 6800.655826] LustreError: 134491:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761252182 with bad export cookie 7292607472342691989
[ 6800.667403] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6811.283681] Lustre: server umount lustre-OST0000 complete
[ 6813.671473] Lustre: 3334:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761252179/real 1761252179]  req@ffff99fb6f6a2300 x1846799703043072/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761252195 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6813.707739] Lustre: 3334:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 6813.717838] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6815.750564] Lustre: server umount lustre-OST0001 complete
[ 6828.927507] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local
[ 6837.019918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6837.022395] Lustre: Skipped 2 previous similar messages
[ 6840.442502] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6843.333729] Lustre: 137726:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6848.031627] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 6853.076496] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6861.692990] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5963 to 0x280000400:5985)
[ 6861.706263] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5963 to 0x240000400:5985)
[ 6864.780618] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6872.017995] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6875.193372] Lustre: 139517:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6911.550639] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 16:44:51 (1761252291)
[ 6950.792964] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 16:45:30 (1761252330)
[ 6994.698783] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 16:46:14 (1761252374)
[ 6996.180438] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs
[ 6997.665578] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 16:46:17 (1761252377)
[ 6999.061201] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs
[ 7000.768610] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 16:46:20 (1761252380)
[ 7014.714021] LustreError: 142366:0:(qsd_reint.c:618:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:0 until pending updates are flushed.
[ 7014.726261] LustreError: 142366:0:(qsd_reint.c:618:qqi_reint_delayed()) Skipped 2 previous similar messages
[ 7080.503617] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 16:47:40 (1761252460)
[ 7120.781353] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 16:48:20 (1761252500)
[ 7240.443855] Lustre: DEBUG MARKER: sanity-quota test_48: @@@@@@ FAIL: QMT: mds1 qid 60000 is not deleted