[ 4575.901564] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 4581.010698] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 4584.098982] LustreError: 96504:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 4585.058402] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4586.805911] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 4588.931568] Lustre: Failing over lustre-MDT0000
[ 4589.082856] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 4589.102141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 4589.112697] Lustre: Skipped 1 previous similar message
[ 4589.378517] LustreError: 96698:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup
[ 4589.389841] LustreError: 96698:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message
[ 4589.622367] Lustre: server umount lustre-MDT0000 complete
[ 4607.361427] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4607.955921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4608.008209] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 4608.870791] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 4609.214795] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 4609.333562] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 4609.380273] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:685 to 0x240000400:705)
[ 4609.384174] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:650 to 0x280000400:673)
[ 4611.834454] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4619.340559] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4620.951241] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4626.190645] Lustre: DEBUG MARKER: (dd_pid=93437, time=0, timeout=600)
[ 4666.052171] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 4670.896308] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 4674.282061] LustreError: 98933:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 4675.264954] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4676.993972] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 4679.096730] Lustre: Failing over lustre-MDT0000
[ 4679.451941] LustreError: 99128:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup
[ 4679.455097] LustreError: 99128:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 4679.599917] Lustre: server umount lustre-MDT0000 complete
[ 4697.059917] Lustre: 3321:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763406281/real 1763406281]  req@ffff8caf26f46d80 x1849060735787136/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763406297 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4697.061462] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4697.087493] Lustre: 3321:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 4698.145942] Lustre: 3323:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763406282/real 1763406282]  req@ffff8caf0949c000 x1849060735787264/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1763406298 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 4698.160384] Lustre: 3323:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 4701.216160] Lustre: 3321:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763406286/real 1763406286]  req@ffff8caf3acf0000 x1849060735787648/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763406302 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4706.281809] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xc6d25b47d180f5b2
[ 4706.402695] Lustre: 3324:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763406291/real 1763406291]  req@ffff8caf0949d880 x1849060735787904/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763406307 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4706.448119] Lustre: 3324:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 4706.969576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4707.050956] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 4707.474683] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 4707.659280] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 4707.718211] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:707 to 0x240000400:737)
[ 4707.720054] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:650 to 0x280000400:705)
[ 4710.623685] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4711.848458] Lustre: 3323:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763406296/real 1763406296]  req@ffff8caf26f46680 x1849060735788416/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763406312 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4711.879797] Lustre: 3323:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 4717.946137] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4719.767324] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4722.155564] LustreError: 3320:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-lwp-OST0001: namespace resource [0x200000006:0x20000:0x0].0x0 (ffff8caf3646d800) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4722.191744] LustreError: 3320:0:(ldlm_resource.c:981:ldlm_resource_complain()) Skipped 5 previous similar messages
[ 4741.448495] Lustre: DEBUG MARKER: (dd_pid=95623, time=14, timeout=600)
[ 4794.459975] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 14:06:34 (1763406394)
[ 4817.044061] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 4819.120949] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 4824.737649] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 4826.630094] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 4859.851906] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 14:07:39 (1763406459)
[ 4880.812744] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 4896.571211] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 4898.168475] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 4899.786972] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 4901.561220] Lustre: DEBUG MARKER: Set quota for 1 times
[ 4904.915773] Lustre: DEBUG MARKER: Set quota for 2 times
[ 4907.923956] Lustre: DEBUG MARKER: Set quota for 3 times
[ 4911.281140] Lustre: DEBUG MARKER: Set quota for 4 times
[ 4914.502903] Lustre: DEBUG MARKER: Set quota for 5 times
[ 4917.887611] Lustre: DEBUG MARKER: Set quota for 6 times
[ 4921.228434] Lustre: DEBUG MARKER: Set quota for 7 times
[ 4924.467815] Lustre: DEBUG MARKER: Set quota for 8 times
[ 4927.693507] Lustre: DEBUG MARKER: Set quota for 9 times
[ 4930.848086] Lustre: DEBUG MARKER: Set quota for 10 times
[ 4978.988440] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 14:09:38 (1763406578)
[ 4993.512418] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 4993.522121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 4993.530627] Lustre: Skipped 4 previous similar messages
[ 4993.539967] Lustre: Skipped 1 previous similar message
[ 4996.880707] LustreError: 105539:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup
[ 4996.886948] LustreError: 105539:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 4997.021041] Lustre: server umount lustre-MDT0000 complete
[ 5000.794604] LustreError: 24625:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763406601 with bad export cookie 14326613728634664370
[ 5000.801892] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5000.895482] Lustre: server umount lustre-OST0000 complete
[ 5005.358841] LustreError: 105945:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup
[ 5005.361855] LustreError: 105945:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages
[ 5005.555628] Lustre: server umount lustre-OST0001 complete
[ 5020.357651] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local
[ 5028.865870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5032.577643] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5035.336392] Lustre: 107407:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 5041.333742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 5047.286077] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5053.538192] LustreError: 107721:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5053.551383] LustreError: 107721:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message
[ 5053.565676] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:740 to 0x240000400:769)
[ 5053.830689] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 5055.395228] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:707 to 0x280000400:737)
[ 5059.393386] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5066.227398] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 5069.709460] Lustre: 109181:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 5093.169397] LustreError: 109982:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup
[ 5093.189386] LustreError: 109982:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages
[ 5093.289101] Lustre: server umount lustre-MDT0000 complete
[ 5096.862842] LustreError: 106971:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763406697 with bad export cookie 14326613728634672147
[ 5096.882640] LustreError: 106971:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 5096.903442] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5107.282781] Lustre: server umount lustre-OST0000 complete
[ 5110.241647] Lustre: 3323:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763406695/real 1763406695]  req@ffff8caf36426300 x1849060735944064/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763406711 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5110.290232] Lustre: 3323:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 5110.919989] LustreError: 110393:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup
[ 5110.922973] LustreError: 110393:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages
[ 5111.025130] Lustre: server umount lustre-OST0001 complete
[ 5124.361638] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local
[ 5130.504924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5133.837344] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5136.194399] Lustre: 111846:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 5141.218623] LustreError: 112161:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5141.257773] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:740 to 0x240000400:801)
[ 5144.458616] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5146.602441] LustreError: 112161:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5151.717337] LustreError: 112160:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5152.214109] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 5152.220376] Lustre: Skipped 1 previous similar message
[ 5153.747932] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:707 to 0x280000400:769)
[ 5159.600692] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5168.348521] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 5172.236890] Lustre: 113616:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 5191.128505] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 14:13:09 (1763406789)
[ 5193.630845] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 5195.710446] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 14:13:15 (1763406795)
[ 5247.783910] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 14:14:07 (1763406847)
[ 5291.738694] Lustre: DEBUG MARKER: Write...
[ 5294.028792] Lustre: DEBUG MARKER: Write out of block quota ...
[ 5345.933400] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 14:15:45 (1763406945)
[ 5352.646243] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 14:15:52 (1763406952)
[ 5362.153260] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 14:16:02 (1763406962)
[ 5370.802168] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 14:16:10 (1763406970)
[ 5379.404844] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 14:16:19 (1763406979)
[ 5436.630043] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 5556.807717] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 5716.423567] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 14:21:55 (1763407315)
[ 5767.715866] Lustre: DEBUG MARKER: Restart...
[ 5770.628513] LustreError: 126372:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup
[ 5770.631746] LustreError: 126372:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages
[ 5770.710567] Lustre: server umount lustre-MDT0000 complete
[ 5774.000770] LustreError: 116479:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763407374 with bad export cookie 14326613728634673680
[ 5774.007698] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5783.598906] LustreError: 126577:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup
[ 5783.610793] LustreError: 126577:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 5783.660367] Lustre: server umount lustre-OST0000 complete
[ 5787.936125] Lustre: 3322:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763407372/real 1763407372]  req@ffff8caf08315880 x1849060736146944/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763407388 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5787.965339] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5787.984766] Lustre: Skipped 1 previous similar message
[ 5788.114138] Lustre: server umount lustre-OST0001 complete
[ 5802.857996] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local
[ 5813.308881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5817.983819] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5821.182750] Lustre: 128245:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 5829.274151] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 5838.430639] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5838.434595] LustreError: 128549:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5838.453456] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:811 to 0x240000400:833)
[ 5843.937945] LustreError: 128549:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5848.775827] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 5850.296168] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:777 to 0x280000400:801)
[ 5855.844424] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5864.250990] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 5867.910843] Lustre: 130024:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 5930.777255] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 14:25:30 (1763407530)
[ 6002.099530] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 14:26:41 (1763407601)
[ 7901.032755] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 14:58:20 (1763409500)
[ 7915.333619] LustreError: 135352:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup
[ 7915.340500] LustreError: 135352:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 7915.477384] Lustre: server umount lustre-MDT0000 complete
[ 7920.110844] LustreError: 127816:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763409520 with bad export cookie 14326613728634680141
[ 7920.122292] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 7929.415271] LustreError: 135555:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup
[ 7929.417800] LustreError: 135555:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 7929.468019] Lustre: server umount lustre-OST0000 complete
[ 7933.666402] Lustre: 3322:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763409518/real 1763409518]  req@ffff8cae14fe1880 x1849060739159424/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763409534 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7933.691224] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 7934.488439] LustreError: 135763:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup
[ 7934.491798] LustreError: 135763:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message
[ 7934.724557] Lustre: server umount lustre-OST0001 complete
[ 7950.885647] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local
[ 7961.677187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 7965.648623] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7968.801753] Lustre: 137220:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 7975.209646] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 7981.134948] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7986.469463] LustreError: 137546:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 7986.488296] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5835 to 0x240000400:5857)
[ 7988.474152] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 7990.149660] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5803 to 0x280000400:5825)
[ 7994.520892] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8002.828792] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 8007.135608] Lustre: 138997:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 8045.265212] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 15:00:45 (1763409645)
[ 8086.078457] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 15:01:25 (1763409685)
[ 8129.613317] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 15:02:09 (1763409729)
[ 8131.680130] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs
[ 8133.760333] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 15:02:13 (1763409733)
[ 8135.629809] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs
[ 8137.602331] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 15:02:17 (1763409737)
[ 8219.017076] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 15:03:38 (1763409818)
[ 8261.268817] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 15:04:21 (1763409861)
[ 8379.506678] Lustre: DEBUG MARKER: sanity-quota test_48: @@@@@@ FAIL: QMT: mds1 qid 60000 is not deleted