[ 4434.013620] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 4437.673666] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 4439.778612] LustreError: 129123:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 4440.368855] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4441.376690] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 4442.618120] Lustre: Failing over lustre-MDT0000
[ 4442.949959] Lustre: server umount lustre-MDT0000 complete
[ 4458.172887] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4458.366589] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 4458.385797] Lustre: Skipped 1 previous similar message
[ 4458.495634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4458.560906] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 4460.426847] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 4460.471905] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 4460.508503] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1063 to 0x240000400:1089)
[ 4460.508579] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:997 to 0x280000400:1025)
[ 4460.972890] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4462.303227] Lustre: 3283:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788905080/real 1788905080]  req@ffff9538cd774700 x1875798354888320/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788905096 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4462.323822] Lustre: 3283:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 4463.586445] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 4466.651655] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4467.566244] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4467.679188] Lustre: 3286:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788905085/real 1788905085]  req@ffff95380a3f2a00 x1875798354888576/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788905101 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4467.703919] Lustre: 3286:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 4471.052680] Lustre: DEBUG MARKER: (dd_pid=120118, time=0, timeout=600)
[ 4472.799152] Lustre: 3285:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788905090/real 1788905090]  req@ffff95380ad1a680 x1875798354889088/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788905106 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4472.815551] Lustre: 3285:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 4506.729312] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 4509.857919] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 4511.565918] LustreError: 132112:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 4512.082470] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4512.974786] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 4514.018268] Lustre: Failing over lustre-MDT0000
[ 4514.302639] Lustre: server umount lustre-MDT0000 complete
[ 4528.730710] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4528.911737] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 4528.921098] Lustre: Skipped 1 previous similar message
[ 4529.014161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4529.071604] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 4530.975122] Lustre: 3283:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788905148/real 1788905148]  req@ffff9537c7d6aa00 x1875798354918784/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788905164 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4531.208406] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4532.111446] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 4532.199890] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 4532.218958] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1091 to 0x240000400:1121)
[ 4532.220145] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:997 to 0x280000400:1057)
[ 4534.242956] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 4534.250785] Lustre: Skipped 1 previous similar message
[ 4536.342335] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4537.082700] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4540.319100] Lustre: 3286:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788905158/real 1788905158]  req@ffff9537c7d69500 x1875798354919552/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788905174 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4540.335107] Lustre: 3286:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 4542.078064] Lustre: DEBUG MARKER: (dd_pid=122641, time=2, timeout=600)
[ 4586.654315] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 18:07:00 (1788905220)
[ 4599.551793] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 4600.225499] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 4602.568802] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 4603.239462] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 4625.637136] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 18:07:39 (1788905259)
[ 4626.728228] LustreError: 3285:0:(qsd_reint.c:633:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:0 until pending updates are flushed.
[ 4635.617762] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 4644.556949] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 4645.088109] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 4645.601572] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 4646.249834] Lustre: DEBUG MARKER: Set quota for 1 times
[ 4648.025833] Lustre: DEBUG MARKER: Set quota for 2 times
[ 4649.846655] Lustre: DEBUG MARKER: Set quota for 3 times
[ 4651.681378] Lustre: DEBUG MARKER: Set quota for 4 times
[ 4653.498714] Lustre: DEBUG MARKER: Set quota for 5 times
[ 4655.265381] Lustre: DEBUG MARKER: Set quota for 6 times
[ 4657.050253] Lustre: DEBUG MARKER: Set quota for 7 times
[ 4658.828191] Lustre: DEBUG MARKER: Set quota for 8 times
[ 4660.671064] Lustre: DEBUG MARKER: Set quota for 9 times
[ 4662.366857] Lustre: DEBUG MARKER: Set quota for 10 times
[ 4664.085290] Lustre: DEBUG MARKER: Set quota for 11 times
[ 4665.953189] Lustre: DEBUG MARKER: Set quota for 12 times
[ 4667.808957] Lustre: DEBUG MARKER: Set quota for 13 times
[ 4669.567530] Lustre: DEBUG MARKER: Set quota for 14 times
[ 4671.354496] Lustre: DEBUG MARKER: Set quota for 15 times
[ 4673.124458] Lustre: DEBUG MARKER: Set quota for 16 times
[ 4674.864577] Lustre: DEBUG MARKER: Set quota for 17 times
[ 4697.084841] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 18:08:50 (1788905330)
[ 4703.199502] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 4703.199723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 4703.203013] Lustre: Skipped 1 previous similar message
[ 4705.089643] Lustre: server umount lustre-MDT0000 complete
[ 4706.432187] LustreError: 5803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905340 with bad export cookie 5457107733828966250
[ 4706.432532] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4706.436724] LustreError: 5803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 4706.475922] Lustre: server umount lustre-OST0000 complete
[ 4707.824770] Lustre: server umount lustre-OST0001 complete
[ 4712.428996] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local
[ 4716.012897] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4717.412820] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4718.379728] Lustre: 142608:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 4720.366703] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 4722.137562] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4724.503336] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 4725.542929] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1123 to 0x240000400:1153)
[ 4725.543211] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1060 to 0x280000400:1089)
[ 4726.374707] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4729.356076] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 4730.416972] Lustre: 144512:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 4738.659434] Lustre: server umount lustre-MDT0000 complete
[ 4739.872493] LustreError: 142159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905373 with bad export cookie 5457107733828988958
[ 4739.876584] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4749.304401] Lustre: server umount lustre-OST0000 complete
[ 4760.785063] Lustre: server umount lustre-OST0001 complete
[ 4765.489274] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local
[ 4768.686060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4770.027907] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4771.065695] Lustre: 147210:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 4775.217540] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4778.919249] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1060 to 0x280000400:1121)
[ 4778.919278] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1123 to 0x240000400:1185)
[ 4779.694527] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4782.641781] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 4783.746220] Lustre: 149118:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 4786.813162] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 18:10:20 (1788905420)
[ 4787.292384] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 4787.810679] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 18:10:21 (1788905421)
[ 4824.002459] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 18:10:57 (1788905457)
[ 4841.895340] Lustre: DEBUG MARKER: Write...
[ 4842.702824] Lustre: DEBUG MARKER: Write out of block quota ...
[ 4875.478204] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 18:11:49 (1788905509)
[ 4877.480727] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 18:11:51 (1788905511)
[ 4881.720972] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 18:11:55 (1788905515)
[ 4884.972179] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 18:11:58 (1788905518)
[ 4888.186359] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 18:12:01 (1788905521)
[ 4931.909370] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 5044.763901] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 5191.067994] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 18:17:04 (1788905824)
[ 5242.702913] Lustre: DEBUG MARKER: Restart...
[ 5250.016701] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5250.027146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5250.032475] Lustre: Skipped 2 previous similar messages
[ 5252.808184] Lustre: server umount lustre-MDT0000 complete
[ 5256.914622] LustreError: 146766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905890 with bad export cookie 5457107733828990477
[ 5256.918518] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5256.921983] LustreError: 146766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 5256.996198] Lustre: server umount lustre-OST0000 complete
[ 5260.983328] Lustre: server umount lustre-OST0001 complete
[ 5274.439597] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local
[ 5283.527106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5283.530331] Lustre: Skipped 2 previous similar messages
[ 5286.835335] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5289.007637] Lustre: 168847:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 5294.114634] LustreError: 169217:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5294.122356] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1195 to 0x240000400:1217)
[ 5294.132652] LustreError: 169217:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages
[ 5297.079400] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5299.182314] LustreError: 169217:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5308.306401] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5308.427930] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1129 to 0x280000400:1153)
[ 5314.920474] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 5317.532426] Lustre: 170770:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 5364.258915] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 18:19:57 (1788905997)
[ 5424.235325] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 18:20:57 (1788906057)
[ 6625.878556] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 18:40:59 (1788907259)
[ 6635.792376] Lustre: server umount lustre-MDT0000 complete
[ 6638.075612] LustreError: 168399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788907271 with bad export cookie 5457107733828996819
[ 6638.081673] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6647.422911] Lustre: server umount lustre-OST0000 complete
[ 6663.995515] Lustre: server umount lustre-OST0001 complete
[ 6684.580534] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local
[ 6697.438846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6697.441956] Lustre: Skipped 2 previous similar messages
[ 6702.253383] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6704.477712] Lustre: 179173:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6710.878579] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 6719.034896] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6723.131825] LustreError: 179534:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6723.196402] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:6219 to 0x240000400:6241)
[ 6728.020821] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6733.317779] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:6155 to 0x280000400:6177)
[ 6734.367592] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6741.654294] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6744.209730] Lustre: 181085:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6778.607740] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 18:43:31 (1788907411)
[ 6817.156528] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 18:44:10 (1788907450)
[ 6860.983721] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 18:44:54 (1788907494)
[ 6862.053445] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs
[ 6863.447172] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 18:44:56 (1788907496)
[ 6864.827550] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs
[ 6866.581749] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 18:44:59 (1788907499)
[ 6950.822556] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 18:46:24 (1788907584)
[ 6985.835868] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 18:46:59 (1788907619)
[ 7040.485309] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 18:47:54 (1788907674)
[ 7047.773254] Lustre: *** cfs_fail_loc=a09, val=0***
[ 7047.774920] Lustre: Skipped 1 previous similar message
[ 7048.778445] Lustre: *** cfs_fail_loc=a09, val=0***
[ 7048.780131] Lustre: Skipped 137 previous similar messages
[ 7050.784275] Lustre: *** cfs_fail_loc=a09, val=0***
[ 7050.785883] Lustre: Skipped 279 previous similar messages
[ 7054.791077] Lustre: *** cfs_fail_loc=a09, val=0***
[ 7054.792877] Lustre: Skipped 527 previous similar messages
[ 7062.794348] Lustre: *** cfs_fail_loc=a09, val=0***
[ 7062.795837] Lustre: Skipped 1203 previous similar messages
[ 7078.803160] Lustre: *** cfs_fail_loc=a09, val=0***
[ 7078.804974] Lustre: Skipped 2149 previous similar messages
[ 7316.960314] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 7316.961887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 7316.965846] Lustre: Skipped 2 previous similar messages
[ 7322.080207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 7322.083664] Lustre: Skipped 2 previous similar messages
[ 7322.267380] Lustre: server umount lustre-MDT0000 complete
[ 7324.024337] LustreError: 184761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788907957 with bad export cookie 5457107733830766286
[ 7324.030405] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 7324.204322] Lustre: server umount lustre-OST0000 complete
[ 7325.993953] Lustre: server umount lustre-OST0001 complete
[ 7328.781626] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid
[ 7332.253663] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local
[ 7336.403763]  vdc: vdc1 vdc9
[ 7340.831116]  vde: vde1 vde9
[ 7345.428491]  vdf: vdf1 vdf9
[ 7351.869714] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local
[ 7355.513512] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000
[ 7355.594805] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[ 7355.633579] Lustre: lustre-MDT0000: new disk, initializing
[ 7355.744270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 7355.765266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[ 7357.405131] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7359.953673] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params
[ 7362.418323] Lustre: lustre-OST0000: new disk, initializing
[ 7362.420450] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[ 7362.423151] Lustre: Skipped 1 previous similar message
[ 7362.457834] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 7364.270552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost
[ 7364.274249] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost]
[ 7364.318553] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400
[ 7364.918219] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7369.448136] Lustre: lustre-OST0001: new disk, initializing
[ 7369.450515] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[ 7369.481201] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 7370.888248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost
[ 7370.893149] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost]
[ 7370.930954] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400
[ 7371.782424] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7376.627234] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 7377.972065] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params
[ 7398.746640] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 18:53:52 (1788908032)
[ 7401.156644] Lustre: DEBUG MARKER: == sanity-quota test 49c: lfs quota long options don't consume an extra argument ========================================================== 18:53:54 (1788908034)
[ 7403.668671] Lustre: DEBUG MARKER: == sanity-quota test 49d: lfs quota -d and --delimiter both work ========================================================== 18:53:57 (1788908037)
[ 7406.191532] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 18:53:59 (1788908039)
[ 7437.539052] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 18:54:31 (1788908071)
[ 7473.334093] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 18:55:06 (1788908106)
[ 7487.726328] Lustre: DEBUG MARKER: rename directory return 255
[ 7514.714873] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 18:55:48 (1788908148)
[ 7533.583300] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 18:56:07 (1788908167)
[ 7557.341744] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 18:56:30 (1788908190)
[ 7598.979039] LustreError: 196156:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-0x0 id:60001 enforced:1 hard:51200 soft:0 granted:51200 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[ 7622.266791] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 18:57:35 (1788908255)
[ 7642.149453] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 18:57:55 (1788908275)
[ 7673.488893] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 18:58:27 (1788908307)
[ 7759.618617] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 18:59:53 (1788908393)
[ 7760.186332] Lustre: DEBUG MARKER: SKIP: sanity-quota test_59 ldiskfs only test
[ 7760.788372] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 18:59:54 (1788908394)
[ 7795.875020] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[ 7796.471276] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 19:00:30 (1788908430)
[ 7814.036742] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[ 7814.580184] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 19:00:48 (1788908448)
[ 7845.228470] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[ 7845.830199] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 19:01:19 (1788908479)
[ 7876.881889] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 19:01:50 (1788908510)
[ 7877.391876] Lustre: DEBUG MARKER: SKIP: sanity-quota test_67 ZFS grants some block space together with inode
[ 7878.003056] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 19:01:51 (1788908511)
[ 7893.168492] LustreError: 220383:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[ 7919.217273] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 19:02:32 (1788908552)
[ 7938.039789] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[ 7939.814136] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[ 8071.665140] LustreError: 3285:0:(qsd_reint.c:633:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:0 until pending updates are flushed.
[ 8073.939767] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 19:05:07 (1788908707)
[ 8110.648540] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 19:05:44 (1788908744)
[ 8133.136324] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 19:06:06 (1788908766)
[ 8134.274284] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71a ZFS grants some block space together with inode
[ 8135.469500] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 19:06:08 (1788908768)
[ 8136.435326] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71b ZFS grants some block space together with inode
[ 8137.450421] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 19:06:10 (1788908770)
[ 8150.101230] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB)
[ 8162.007443] Lustre: DEBUG MARKER: Write...
[ 8163.506512] Lustre: DEBUG MARKER: Write out of block quota ...
[ 8213.190766] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 19:07:26 (1788908846)
[ 8233.255779] Lustre: DEBUG MARKER: set to use default quota
[ 8234.325518] Lustre: DEBUG MARKER: set default quota
[ 8235.347426] Lustre: DEBUG MARKER: get default quota
[ 8239.140213] Lustre: DEBUG MARKER: Test not out of quota
[ 8241.992646] Lustre: DEBUG MARKER: Test out of quota
[ 8249.044167] Lustre: DEBUG MARKER: Increase default quota
[ 8269.211912] Lustre: DEBUG MARKER: Set quota to override default quota
[ 8269.241800] LustreError: 196155:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45063 time:1789513702 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[ 8275.623394] Lustre: DEBUG MARKER: Set to use default quota again
[ 8290.807107] Lustre: DEBUG MARKER: Cleanup
[ 8346.910876] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 19:09:40 (1788908980)
[ 8364.501651] Lustre: DEBUG MARKER: set default quota for qpool1
[ 8365.492787] Lustre: DEBUG MARKER: Write from user that hasn't lqe
[ 8402.363315] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 19:10:35 (1788909035)
[ 8454.700924] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 19:11:28 (1788909088)
[ 8502.275756] Lustre: DEBUG MARKER: Write...
[ 8504.049199] Lustre: DEBUG MARKER: Write out of block quota ...
[ 8576.150318] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 19:13:29 (1788909209)
[ 8605.428385] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 19:13:59 (1788909239)
[ 8608.369954] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 19:14:01 (1788909241)
[ 8609.323562] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78A fallocate not supported
[ 8609.992996] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 19:14:03 (1788909243)
[ 8611.007912] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78a fallocate not supported
[ 8611.699665] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 19:14:05 (1788909245)
[ 8621.376739] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 19:14:15 (1788909255)
[ 8621.984546] Lustre: DEBUG MARKER: SKIP: sanity-quota test_80 ZFS grants some block space together with inode
[ 8622.558925] Lustre: DEBUG MARKER: == sanity-quota test 81: Race qmt_start_pool_recalc with qmt_pool_free ========================================================== 19:14:16 (1788909256)
[ 8632.485691] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[ 8635.580396] LustreError: 246148:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 sleeping for 10000ms
[ 8640.992477] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 8640.996719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 8641.002470] Lustre: Skipped 1 previous similar message
[ 8645.679177] LustreError: 246148:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 awake
[ 8645.857706] Lustre: server umount lustre-MDT0000 complete
[ 8649.796640] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 8650.050370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 8650.116895] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:27 to 0x240000400:65)
[ 8650.116907] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:27 to 0x280000400:65)
[ 8651.953844] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8660.262038] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[ 8660.265082] Lustre: Skipped 1 previous similar message
[ 8681.221889] Lustre: DEBUG MARKER: == sanity-quota test 82: verify more than 8 qids for single operation ========================================================== 19:15:14 (1788909314)
[ 8702.687764] Lustre: DEBUG MARKER: == sanity-quota test 83: Setting default quota shouldn't affect grace time ========================================================== 19:15:36 (1788909336)
[ 8724.599961] Lustre: DEBUG MARKER: == sanity-quota test 84: Reset quota should fix the insane granted quota ========================================================== 19:15:57 (1788909357)
[ 8757.537941] Lustre: *** cfs_fail_loc=a08, val=0***
[ 8757.540239] Lustre: Skipped 3699 previous similar messages
[ 8757.543890] Lustre: *** cfs_fail_loc=a08, val=0***
[ 8820.188395] Lustre: DEBUG MARKER: == sanity-quota test 85: do not hung at write with the least_qunit ========================================================== 19:17:33 (1788909453)
[ 8880.799573] Lustre: DEBUG MARKER: == sanity-quota test 86: Pre-acquired quota should be released if quota is over limit ========================================================== 19:18:34 (1788909514)
[ 8922.179206] LustreError: 248752:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1789514355 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[ 9002.537257] LustreError: 255467:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1789514436 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[ 9093.623848] LustreError: 248752:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:1000 enforced:1 hard:2048 soft:2048 granted:65536 time:1789514527 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[ 9100.570383] Lustre: DEBUG MARKER: sanity-quota test_86: @@@@@@ FAIL: succeed to create files, expect failed