[ 3996.231289] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 4005.540663] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 4015.537516] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4017.513847] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 4019.645373] Lustre: Failing over lustre-MDT0000
[ 4020.013362] Lustre: server umount lustre-MDT0000 complete
[ 4020.515890] LustreError: lustre-MDT0000-lwp-OST0000: operation quota_acquire to node 0@lo failed: rc = -107
[ 4020.535450] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 4020.559380] Lustre: Skipped 1 previous similar message
[ 4020.578075] LustreError: 40919:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 4020.606213] LustreError: 40919:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages
[ 4037.986123] Lustre: 3653:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782484441/real 1782484441]  req@ffff8a7244b34700 x1869066356942848/t0(0) o400->MGC192.168.202.124@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1782484457 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4038.018120] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4043.016801] LDISKFS-fs (dm-0): 7 truncates cleaned up
[ 4043.023165] LDISKFS-fs (dm-0): recovery complete
[ 4043.064335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4048.352745] LustreError: 3651:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight  req@ffff8a7376e11500 x1869066356953216/t0(0) o250->MGC192.168.202.124@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4048.643989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4048.680605] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 4049.739155] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 4053.801452] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4053.993351] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 4054.016547] Lustre: Skipped 1 previous similar message
[ 4054.071111] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted.
[ 4054.124476] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1423 to 0x280000401:1441)
[ 4054.136426] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1387 to 0x2c0000401:1409)
[ 4063.500763] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4065.646943] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4072.327518] Lustre: DEBUG MARKER: (dd_pid=130046, time=0, timeout=600)
[ 4109.764565] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 4116.109712] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 4124.860918] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4127.003302] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 4129.283671] Lustre: Failing over lustre-MDT0000
[ 4129.728951] Lustre: server umount lustre-MDT0000 complete
[ 4130.788332] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 4147.172572] Lustre: 3653:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782484551/real 1782484551]  req@ffff8a7248ea3480 x1869066357013504/t0(0) o400->MGC192.168.202.124@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1782484567 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4147.198511] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4152.120596] LustreError: 6524:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 4152.145765] LustreError: 6524:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 57 previous similar messages
[ 4154.406314] LDISKFS-fs (dm-0): 3 truncates cleaned up
[ 4154.408475] LDISKFS-fs (dm-0): recovery complete
[ 4154.423452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4156.576190] LustreError: 148796:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0)
[ 4156.589387] LustreError: 148796:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list  req@ffff8a7381f2b480 x1869066357019520/t0(0) o250->MGC192.168.202.124@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1782484577 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4156.622967] LustreError: 148796:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out.
[ 4157.420606] LustreError: 3651:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight  req@ffff8a724eaf6680 x1869066357023360/t0(0) o250->MGC192.168.202.124@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4157.712212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4157.754086] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 4158.349157] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 4162.669937] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4163.043726] LustreError: 3651:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-lwp-MDT0001: namespace resource [0x200000006:0x10000:0x0].0x0 (ffff8a737f7d6c00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4163.079492] LustreError: 3651:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 5 previous similar messages
[ 4163.112220] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted.
[ 4163.162049] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1387 to 0x2c0000401:1441)
[ 4163.162574] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1443 to 0x280000401:1473)
[ 4173.402575] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4175.398995] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4181.434683] Lustre: DEBUG MARKER: (dd_pid=132577, time=0, timeout=600)
[ 4226.964972] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 10:37:25 (1782484645)
[ 4276.709856] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 10:38:15 (1782484695)
[ 4295.948431] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 4303.781984] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 4304.951197] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 4306.175491] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 4307.486654] Lustre: DEBUG MARKER: Set quota for 1 times
[ 4310.268051] Lustre: DEBUG MARKER: Set quota for 2 times
[ 4312.761332] Lustre: DEBUG MARKER: Set quota for 3 times
[ 4315.225365] Lustre: DEBUG MARKER: Set quota for 4 times
[ 4317.739988] Lustre: DEBUG MARKER: Set quota for 5 times
[ 4320.096297] Lustre: DEBUG MARKER: Set quota for 6 times
[ 4322.475728] Lustre: DEBUG MARKER: Set quota for 7 times
[ 4324.774138] Lustre: DEBUG MARKER: Set quota for 8 times
[ 4327.100754] Lustre: DEBUG MARKER: Set quota for 9 times
[ 4329.474603] Lustre: DEBUG MARKER: Set quota for 10 times
[ 4331.916159] Lustre: DEBUG MARKER: Set quota for 11 times
[ 4334.558674] Lustre: DEBUG MARKER: Set quota for 12 times
[ 4368.577828] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 10:39:48 (1782484788)
[ 4378.081593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 4378.081673] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 4378.084343] Lustre: Skipped 1 previous similar message
[ 4383.201066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 4383.204729] Lustre: Skipped 3 previous similar messages
[ 4383.417517] Lustre: server umount lustre-MDT0000 complete
[ 4385.327721] LustreError: 6511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782484805 with bad export cookie 17669284484888837342
[ 4385.329702] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4385.340131] LustreError: 6511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 4385.492139] Lustre: server umount lustre-MDT0001 complete
[ 4397.585253] Lustre: server umount lustre-OST0000 complete
[ 4408.672280] Lustre: server umount lustre-OST0001 complete
[ 4416.207232] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local
[ 4421.875852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4422.135509] LustreError: 158078:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 4422.144143] LustreError: 158078:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages
[ 4422.198273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4424.439114] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4429.621106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4429.839243] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 4432.361667] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4434.041606] Lustre: 159230:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 4437.589115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 4437.775917] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 4440.711827] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4445.046109] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 4447.269660] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1475 to 0x280000401:1505)
[ 4447.274769] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1444 to 0x2c0000401:1473)
[ 4447.276703] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:97)
[ 4447.843853] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4452.221492] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 4454.094608] Lustre: 161106:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 4465.634528] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 4465.639534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 4467.681971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 4467.685096] Lustre: Skipped 3 previous similar messages
[ 4471.494901] Lustre: server umount lustre-MDT0000 complete
[ 4474.786760] LustreError: 161358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782484895 with bad export cookie 17669284484888846134
[ 4474.790956] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4474.800603] LustreError: 161358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 4475.078898] Lustre: server umount lustre-MDT0001 complete
[ 4488.924908] Lustre: server umount lustre-OST0000 complete
[ 4494.254427] Lustre: 3653:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782484898/real 1782484898]  req@ffff8a724320bb80 x1869066357272448/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782484914 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4495.970608] Lustre: server umount lustre-OST0001 complete
[ 4524.727030] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local
[ 4541.772326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4542.816507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4542.819993] Lustre: Skipped 1 previous similar message
[ 4549.971937] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4559.773910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4560.382726] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 4565.401322] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4568.425840] Lustre: 164946:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 4574.541209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 4580.880876] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4588.204472] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1475 to 0x280000401:1537)
[ 4588.908183] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 4594.177691] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1444 to 0x2c0000401:1505)
[ 4594.255718] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:129)
[ 4594.606300] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4601.557932] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 4604.900635] Lustre: 166814:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 4614.434237] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 10:43:53 (1782485033)
[ 4615.791347] Lustre: DEBUG MARKER: OST0_SIZE: 3605396 required: 6144
[ 4617.262347] Lustre: DEBUG MARKER: run for 4MB test file
[ 4626.858494] Lustre: DEBUG MARKER: User quota (limit: 4 MB)
[ 4632.076606] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT
[ 4633.448332] Lustre: DEBUG MARKER: Write half of file
[ 4635.136966] Lustre: DEBUG MARKER: Write out of block quota ...
[ 4636.621439] Lustre: DEBUG MARKER: Step1: done
[ 4638.019792] Lustre: DEBUG MARKER: Step2: rewrite should succeed
[ 4639.497613] Lustre: DEBUG MARKER: Step2: done
[ 4663.576696] Lustre: DEBUG MARKER: OST0_SIZE: 3605396 required: 61440
[ 4665.016480] Lustre: DEBUG MARKER: run for 40MB test file
[ 4674.166389] Lustre: DEBUG MARKER: User quota (limit: 40 MB)
[ 4679.187373] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT
[ 4680.336868] Lustre: DEBUG MARKER: Write half of file
[ 4682.354822] Lustre: DEBUG MARKER: Write out of block quota ...
[ 4684.718598] Lustre: DEBUG MARKER: Step1: done
[ 4686.076696] Lustre: DEBUG MARKER: Step2: rewrite should succeed
[ 4687.312304] Lustre: DEBUG MARKER: Step2: done
[ 4730.404760] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 10:45:49 (1782485149)
[ 4767.137870] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 10:46:26 (1782485186)
[ 4797.820949] Lustre: DEBUG MARKER: Write...
[ 4799.582753] Lustre: DEBUG MARKER: Write out of block quota ...
[ 4844.401509] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 10:47:43 (1782485263)
[ 4851.190684] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 10:47:50 (1782485270)
[ 4860.923516] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 10:47:59 (1782485279)
[ 4870.646821] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 10:48:09 (1782485289)
[ 4878.536313] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 10:48:17 (1782485297)
[ 4928.019798] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 5044.672907] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 5173.082242] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 10:53:11 (1782485591)
[ 5218.959634] Lustre: DEBUG MARKER: Restart...
[ 5223.907606] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5223.915951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5223.924553] Lustre: Skipped 18 previous similar messages
[ 5223.939773] Lustre: Skipped 3 previous similar messages
[ 5229.025279] LustreError: 163805:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5229.035780] LustreError: 163805:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages
[ 5229.102940] Lustre: server umount lustre-MDT0000 complete
[ 5232.037865] LustreError: 165324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782485652 with bad export cookie 17669284484888848437
[ 5232.040176] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5232.050641] LustreError: 165324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 5232.299291] Lustre: server umount lustre-MDT0001 complete
[ 5245.967266] Lustre: server umount lustre-OST0000 complete
[ 5250.020032] Lustre: 3652:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782485654/real 1782485654]  req@ffff8a7379162300 x1869066357699840/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782485670 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5250.372520] Lustre: server umount lustre-OST0001 complete
[ 5268.531400] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local
[ 5282.348178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5283.256979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5283.264604] Lustre: Skipped 2 previous similar messages
[ 5291.010300] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5307.922071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5309.205067] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 5318.425816] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5325.545712] Lustre: 191436:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 5336.436189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 5337.137769] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 5345.011717] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5352.498748] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1549 to 0x280000401:1569)
[ 5355.232143] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 5355.588861] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 5360.634631] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1513 to 0x2c0000401:1537)
[ 5360.635879] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:161)
[ 5363.058891] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5373.238913] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 5377.636939] Lustre: 193310:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 5446.173411] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 10:57:44 (1782485864)
[ 5503.652173] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 10:58:41 (1782485921)
[ 6913.317389] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 11:22:12 (1782487332)
[ 6921.697486] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 6921.700045] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6921.703981] Lustre: Skipped 2 previous similar messages
[ 6921.705874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6922.212761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6922.216209] Lustre: Skipped 3 previous similar messages
[ 6924.199839] Lustre: server umount lustre-MDT0000 complete
[ 6926.070254] LustreError: 193312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782487346 with bad export cookie 17669284484888857481
[ 6926.075293] LustreError: 193312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message
[ 6926.077885] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6926.281266] Lustre: server umount lustre-MDT0001 complete
[ 6935.520365] LustreError: 3653:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED  req@ffff8a7246c05500 x1869066361024896/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 6935.537071] LustreError: 3653:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8  qsd:lustre-OST0000 qtype:usr lqe: ffff8a7244df9980 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 1596 qunit:0 qtune:0 edquot:0 default:no revoke:0
[ 6938.264263] Lustre: server umount lustre-OST0000 complete
[ 6950.147970] Lustre: server umount lustre-OST0001 complete
[ 6957.582470] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local
[ 6963.076918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 6963.284344] LustreError: 200610:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6963.291673] LustreError: 200610:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages
[ 6963.333138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6965.195983] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6968.982748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 6969.201149] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 6971.076590] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6972.493141] Lustre: 201753:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6975.811719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 6978.985229] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6983.194944] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 6983.338233] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6983.340303] Lustre: Skipped 1 previous similar message
[ 6985.925793] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6988.454514] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:6572 to 0x280000401:6593)
[ 6988.461347] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:193)
[ 6988.471086] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:6538 to 0x2c0000401:6561)
[ 6995.006793] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6996.698178] Lustre: 203626:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 7015.137593] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 11:23:54 (1782487434)
[ 7031.071334] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 11:24:10 (1782487450)
[ 7052.038944] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 11:24:31 (1782487471)
[ 7071.887989] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 11:24:51 (1782487491)
[ 7097.400198] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 11:25:16 (1782487516)
[ 7144.279486] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 11:26:03 (1782487563)
[ 7161.581780] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 11:26:21 (1782487581)
[ 7168.332987] LustreError: 202115:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -3, flags:0x1  qsd:lustre-OST0001 qtype:usr lqe: ffff8a724b292180 id:60000 enforced:1 granted: 0 pending:0 waiting:1032 req:1 usage: 0 qunit:0 qtune:0 edquot:0 default:no revoke:0
[ 7168.340227] LustreError: 202115:0:(qsd_handler.c:773:qsd_op_begin0()) $$$ ID isn't enforced on master, it probably due to a legeal race, if this message is showing up constantly, there could be some inconsistence between master & slave, and quota reintegration needs be re-triggered.  qsd:lustre-OST0001 qtype:usr lqe: ffff8a724b292180 id:60000 enforced:1 granted: 0 pending:0 waiting:0 req:0 usage: 0 qunit:0 qtune:0 edquot:0 default:no revoke:0
[ 7183.706137] LustreError: 202114:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -3, flags:0x1  qsd:lustre-OST0000 qtype:grp lqe: ffff8a72420e1080 id:60000 enforced:1 granted: 0 pending:0 waiting:1032 req:1 usage: 0 qunit:0 qtune:0 edquot:0 default:no revoke:0
[ 7183.718138] LustreError: 202114:0:(qsd_handler.c:298:qsd_req_completion()) Skipped 1 previous similar message
[ 7183.724925] LustreError: 202114:0:(qsd_handler.c:773:qsd_op_begin0()) $$$ ID isn't enforced on master, it probably due to a legeal race, if this message is showing up constantly, there could be some inconsistence between master & slave, and quota reintegration needs be re-triggered.  qsd:lustre-OST0000 qtype:grp lqe: ffff8a72420e1080 id:60000 enforced:1 granted: 0 pending:0 waiting:0 req:0 usage: 0 qunit:0 qtune:0 edquot:0 default:no revoke:0
[ 7183.749615] LustreError: 202114:0:(qsd_handler.c:773:qsd_op_begin0()) Skipped 2 previous similar messages
[ 7246.819262] LustreError: 3653:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -3, flags:0x8  qsd:lustre-OST0000 qtype:grp lqe: ffff8a72420e1080 id:60000 enforced:1 granted: 0 pending:0 waiting:0 req:1 usage: 1024 qunit:0 qtune:0 edquot:0 default:no revoke:0
[ 7246.828299] LustreError: 3653:0:(qsd_handler.c:298:qsd_req_completion()) Skipped 1 previous similar message
[ 7262.828618] Lustre: DEBUG MARKER: sanity-quota test_48: @@@@@@ FAIL: QMT: mds1 qid 60000 is not deleted