[ 4035.434761] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 4038.438439] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 4043.710648] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4044.907916] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 4046.395871] Lustre: Failing over lustre-MDT0000
[ 4046.674675] LustreError: 138747:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup
[ 4046.681850] LustreError: 138747:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages
[ 4046.789915] Lustre: server umount lustre-MDT0000 complete
[ 4047.842306] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 4047.846672] LustreError: 21535:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 4047.854782] LustreError: 21535:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages
[ 4064.228445] Lustre: 6809:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767323945/real 1767323945]  req@ffff9a358743d880 x1853169392215168/t0(0) o400->MGC192.168.202.140@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1767323961 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4064.253369] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4067.443442] LDISKFS-fs (dm-0): 7 truncates cleaned up
[ 4067.445159] LDISKFS-fs (dm-0): recovery complete
[ 4067.455234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4074.471949] LustreError: 6808:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight  req@ffff9a3687e2ea00 x1853169392223616/t0(0) o250->MGC192.168.202.140@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4074.590849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4074.611882] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 4075.853612] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 4076.704221] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4079.587712] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[ 4079.591747] Lustre: Skipped 1 previous similar message
[ 4079.613832] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted.
[ 4079.635096] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:936 to 0x280000401:961)
[ 4079.635275] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:932 to 0x2c0000401:961)
[ 4081.428619] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4082.363564] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4085.817994] Lustre: DEBUG MARKER: (dd_pid=111818, time=0, timeout=600)
[ 4101.222642] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 4104.133949] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 4108.598837] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4109.626660] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 4110.834610] Lustre: Failing over lustre-MDT0000
[ 4111.095310] LustreError: 141931:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup
[ 4111.099142] LustreError: 141931:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages
[ 4111.156099] Lustre: server umount lustre-MDT0000 complete
[ 4115.430611] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 4115.430731] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 4115.438308] Lustre: Skipped 6 previous similar messages
[ 4131.500985] LDISKFS-fs (dm-0): 2 truncates cleaned up
[ 4131.503418] LDISKFS-fs (dm-0): recovery complete
[ 4131.508160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4131.576652] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4131.764964] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4134.065711] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4136.962873] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:932 to 0x2c0000401:993)
[ 4136.966366] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:963 to 0x280000401:993)
[ 4138.938029] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4139.843675] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4143.348504] Lustre: DEBUG MARKER: (dd_pid=114097, time=0, timeout=600)
[ 4164.522361] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 22:21:01 (1767324061)
[ 4195.264845] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 22:21:31 (1767324091)
[ 4208.941712] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 4215.268546] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 4216.225266] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 4217.180955] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 4218.330707] Lustre: DEBUG MARKER: Set quota for 1 times
[ 4220.758800] Lustre: DEBUG MARKER: Set quota for 2 times
[ 4223.369738] Lustre: DEBUG MARKER: Set quota for 3 times
[ 4226.014960] Lustre: DEBUG MARKER: Set quota for 4 times
[ 4228.212691] Lustre: DEBUG MARKER: Set quota for 5 times
[ 4230.675883] Lustre: DEBUG MARKER: Set quota for 6 times
[ 4232.884298] Lustre: DEBUG MARKER: Set quota for 7 times
[ 4235.040470] Lustre: DEBUG MARKER: Set quota for 8 times
[ 4237.420906] Lustre: DEBUG MARKER: Set quota for 9 times
[ 4239.786097] Lustre: DEBUG MARKER: Set quota for 10 times
[ 4242.097409] Lustre: DEBUG MARKER: Set quota for 11 times
[ 4244.353844] Lustre: DEBUG MARKER: Set quota for 12 times
[ 4246.436940] Lustre: DEBUG MARKER: Set quota for 13 times
[ 4276.061881] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 22:22:52 (1767324172)
[ 4285.415573] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 4285.416097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 4285.420473] Lustre: Skipped 2 previous similar messages
[ 4290.527682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 4290.530110] Lustre: Skipped 3 previous similar messages
[ 4295.648647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 4295.650563] Lustre: Skipped 3 previous similar messages
[ 4299.231140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck?
[ 4299.288077] LustreError: 149575:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup
[ 4299.295359] LustreError: 149575:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages
[ 4299.353451] Lustre: server umount lustre-MDT0000 complete
[ 4301.555184] LustreError: 14428:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767324198 with bad export cookie 15296755691746513174
[ 4301.556901] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4301.561462] LustreError: 14428:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages
[ 4301.815525] Lustre: server umount lustre-MDT0001 complete
[ 4315.159454] Lustre: server umount lustre-OST0000 complete
[ 4326.383477] Lustre: server umount lustre-OST0001 complete
[ 4339.121687] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local
[ 4348.950554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4349.186441] LustreError: 152122:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 4349.196359] LustreError: 152122:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 55 previous similar messages
[ 4349.258270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4351.921853] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4355.348655] Lustre: 152682:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 4362.207940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 4362.438993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 4365.053877] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4368.780223] Lustre: 153774:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 4374.996435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 4375.194636] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 4378.299604] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4385.683388] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 4385.805726] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 4387.879106] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97)
[ 4389.142910] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4395.049851] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:996 to 0x2c0000401:1025)
[ 4395.050926] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:995 to 0x280000401:1025)
[ 4398.993305] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 4401.009646] Lustre: 156011:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 4419.039921] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 4419.043695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 4424.607418] Lustre: server umount lustre-MDT0000 complete
[ 4426.995789] LustreError: 153241:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767324323 with bad export cookie 15296755691746522575
[ 4426.996532] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4427.007766] LustreError: 153241:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 4427.189737] Lustre: server umount lustre-MDT0001 complete
[ 4441.566380] Lustre: server umount lustre-OST0000 complete
[ 4454.453357] Lustre: server umount lustre-OST0001 complete
[ 4467.738461] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local
[ 4477.617435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4478.019639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4480.789670] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4484.610979] Lustre: 160012:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 4490.101671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 4492.666204] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4495.914657] Lustre: 161097:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 4502.489803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 4503.718843] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:995 to 0x280000401:1057)
[ 4506.476744] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4513.941399] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 4514.095231] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 4514.100252] Lustre: Skipped 2 previous similar messages
[ 4515.117488] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:129)
[ 4515.139817] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:996 to 0x2c0000401:1057)
[ 4518.020619] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4523.597173] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 4525.891090] Lustre: 163339:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 4536.712677] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 22:27:13 (1767324433)
[ 4537.707919] Lustre: DEBUG MARKER: OST0_SIZE: 3605396 required: 6144
[ 4538.597809] Lustre: DEBUG MARKER: run for 4MB test file
[ 4546.536927] Lustre: DEBUG MARKER: User quota (limit: 4 MB)
[ 4549.356679] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT
[ 4550.219198] Lustre: DEBUG MARKER: Write half of file
[ 4551.246521] Lustre: DEBUG MARKER: Write out of block quota ...
[ 4552.263583] Lustre: DEBUG MARKER: Step1: done
[ 4553.041380] Lustre: DEBUG MARKER: Step2: rewrite should succeed
[ 4554.133649] Lustre: DEBUG MARKER: Step2: done
[ 4567.874968] Lustre: DEBUG MARKER: OST0_SIZE: 3605396 required: 61440
[ 4568.693675] Lustre: DEBUG MARKER: run for 40MB test file
[ 4575.658893] Lustre: DEBUG MARKER: User quota (limit: 40 MB)
[ 4578.284632] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT
[ 4579.182775] Lustre: DEBUG MARKER: Write half of file
[ 4581.043422] Lustre: DEBUG MARKER: Write out of block quota ...
[ 4582.750915] Lustre: DEBUG MARKER: Step1: done
[ 4583.665592] Lustre: DEBUG MARKER: Step2: rewrite should succeed
[ 4584.590305] Lustre: DEBUG MARKER: Step2: done
[ 4617.971076] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 22:28:34 (1767324514)
[ 4644.075126] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 22:29:00 (1767324540)
[ 4665.534236] Lustre: DEBUG MARKER: Write...
[ 4666.867912] Lustre: DEBUG MARKER: Write out of block quota ...
[ 4699.857747] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 22:29:56 (1767324596)
[ 4702.841579] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 22:29:59 (1767324599)
[ 4708.430955] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 22:30:05 (1767324605)
[ 4712.922767] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 22:30:09 (1767324609)
[ 4717.771636] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 22:30:13 (1767324613)
[ 4750.705714] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 4823.168164] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 4904.218234] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 22:33:20 (1767324800)
[ 4928.909673] Lustre: DEBUG MARKER: Restart...
[ 4935.136890] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 4935.141720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 4935.152380] Lustre: Skipped 11 previous similar messages
[ 4935.154784] Lustre: Skipped 6 previous similar messages
[ 4937.296413] LustreError: 182906:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 4937.299071] LustreError: 182906:0:(obd_class.h:479:obd_check_dev()) Skipped 51 previous similar messages
[ 4937.370872] Lustre: server umount lustre-MDT0000 complete
[ 4939.522211] LustreError: 159440:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767324836 with bad export cookie 15296755691746524899
[ 4939.524261] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4939.528320] LustreError: 159440:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages
[ 4939.697148] Lustre: server umount lustre-MDT0001 complete
[ 4953.843700] Lustre: server umount lustre-OST0000 complete
[ 4966.431642] Lustre: server umount lustre-OST0001 complete
[ 4978.390156] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local
[ 4988.277457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4988.575577] LustreError: 185453:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 4988.587324] LustreError: 185453:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages
[ 4988.638773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4991.109717] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4994.527298] Lustre: 186006:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 5001.322978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 5003.994365] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5013.470641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 5014.754799] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1068 to 0x280000401:1089)
[ 5017.835971] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5026.020755] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 5027.241685] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:161)
[ 5027.247597] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1066 to 0x2c0000401:1089)
[ 5029.867416] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5035.144859] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 5037.257360] Lustre: 189327:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 5037.262523] Lustre: 189327:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 1 previous similar message
[ 5073.612176] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 22:36:10 (1767324970)
[ 5107.453325] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 22:36:44 (1767325004)
[ 5917.752946] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 22:50:14 (1767325814)
[ 5923.299363] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 5923.302126] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5923.302836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5923.302840] Lustre: Skipped 1 previous similar message
[ 5923.322427] Lustre: Skipped 3 previous similar messages
[ 5928.143708] LustreError: 194583:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 5928.145866] LustreError: 194583:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages
[ 5928.208670] Lustre: server umount lustre-MDT0000 complete
[ 5928.416527] LustreError: 185453:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5928.425024] LustreError: 185453:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages
[ 5929.792513] LustreError: 185432:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767325826 with bad export cookie 15296755691746533824
[ 5929.792864] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5929.797926] LustreError: 185432:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 5929.916041] Lustre: server umount lustre-MDT0001 complete
[ 5942.831427] Lustre: server umount lustre-OST0000 complete
[ 5954.925161] Lustre: server umount lustre-OST0001 complete
[ 5963.836161] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local
[ 5971.219759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5971.496099] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5971.499610] Lustre: Skipped 3 previous similar messages
[ 5973.165980] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5975.599982] Lustre: 197681:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 5980.564501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 5982.677328] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5985.476698] Lustre: 198768:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 5990.359523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 5990.581078] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 5990.583945] Lustre: Skipped 1 previous similar message
[ 5991.587387] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:193)
[ 5993.315614] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5996.707707] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:6091 to 0x280000401:6113)
[ 5998.777270] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 6000.880261] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:6091 to 0x2c0000401:6113)
[ 6001.432512] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6005.706797] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6007.300863] Lustre: 201001:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6025.448293] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 22:52:02 (1767325922)
[ 6039.452600] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 22:52:16 (1767325936)
[ 6054.296936] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 22:52:31 (1767325951)
[ 6070.997745] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 22:52:47 (1767325967)
[ 6088.444661] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 22:53:05 (1767325985)
[ 6124.189703] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 22:53:41 (1767326021)
[ 6135.773964] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 22:53:52 (1767326032)
[ 6161.557788] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 22:54:18 (1767326058)
[ 6163.975609] Lustre: *** cfs_fail_loc=a09, val=0***
[ 6163.978375] Lustre: Skipped 1 previous similar message
[ 6164.988891] Lustre: *** cfs_fail_loc=a09, val=0***
[ 6164.990033] Lustre: Skipped 119 previous similar messages
[ 6166.995293] Lustre: *** cfs_fail_loc=a09, val=0***
[ 6166.996774] Lustre: Skipped 313 previous similar messages
[ 6171.003444] Lustre: *** cfs_fail_loc=a09, val=0***
[ 6171.005289] Lustre: Skipped 569 previous similar messages
[ 6179.016228] Lustre: *** cfs_fail_loc=a09, val=0***
[ 6179.018009] Lustre: Skipped 955 previous similar messages
[ 6195.020576] Lustre: *** cfs_fail_loc=a09, val=0***
[ 6195.022104] Lustre: Skipped 1825 previous similar messages
[ 6227.034058] Lustre: *** cfs_fail_loc=a09, val=0***
[ 6227.037014] Lustre: Skipped 3085 previous similar messages
[ 6388.193654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6388.195973] Lustre: Skipped 5 previous similar messages
[ 6393.377799] Lustre: server umount lustre-MDT0000 complete
[ 6396.737565] LustreError: 206224:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767326293 with bad export cookie 15296755691748293764
[ 6396.741328] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6396.747152] LustreError: 206224:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 6398.433044] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping)
[ 6398.436291] Lustre: Skipped 1 previous similar message
[ 6406.561394] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping)
[ 6406.563724] Lustre: Skipped 3 previous similar messages
[ 6411.231100] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck?
[ 6411.318715] Lustre: server umount lustre-MDT0001 complete
[ 6415.984819] Lustre: server umount lustre-OST0000 complete
[ 6419.418364] Lustre: server umount lustre-OST0001 complete
[ 6424.740421] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_hostid
[ 6427.560617] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local
[ 6436.054543] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[ 6436.156638] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[ 6437.575050] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[ 6438.637100] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[ 6444.336688] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local
[ 6450.942543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[ 6450.967284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 6451.061751] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000
[ 6451.074135] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[ 6451.109838] Lustre: lustre-MDT0000: new disk, initializing
[ 6451.139805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6451.142466] Lustre: Skipped 1 previous similar message
[ 6451.149552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[ 6452.557934] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6455.854761] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params
[ 6460.742812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[ 6460.769256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 6460.803812] Lustre: 219929:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17
[ 6460.815845] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
[ 6460.817785] Lustre: Skipped 1 previous similar message
[ 6460.853346] Lustre: lustre-MDT0001: new disk, initializing
[ 6460.888027] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt
[ 6460.893941] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt]
[ 6462.531189] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6470.949409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[ 6470.974371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 6471.047052] Lustre: lustre-OST0000: new disk, initializing
[ 6471.048804] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[ 6472.878263] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost
[ 6472.881996] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost]
[ 6472.905578] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401
[ 6473.265976] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6480.006181] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[ 6480.038474] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 6480.087708] Lustre: lustre-OST0001: new disk, initializing
[ 6480.090120] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[ 6481.530340] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost
[ 6481.533339] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost]
[ 6481.544083] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401
[ 6482.181777] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6486.861370] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6488.159600] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params
[ 6503.806542] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 23:00:00 (1767326400)
[ 6506.182996] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 23:00:02 (1767326402)
[ 6522.020411] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 23:00:18 (1767326418)
[ 6542.388695] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 23:00:39 (1767326439)
[ 6549.309087] Lustre: DEBUG MARKER: rename directory return 255
[ 6568.451173] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 23:01:05 (1767326465)
[ 6576.373859] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 23:01:13 (1767326473)
[ 6587.448318] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 23:01:24 (1767326484)
[ 6617.998848] LustreError: 231606:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-0x0 id:60001 enforced:1 hard:51200 soft:0 granted:51200 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[ 6629.144450] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 23:02:06 (1767326526)
[ 6638.007728] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 23:02:14 (1767326534)
[ 6649.807230] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 23:02:26 (1767326546)
[ 6701.192771] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 23:03:18 (1767326598)
[ 6706.655553] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 6706.657792] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6706.661047] Lustre: Skipped 6 previous similar messages
[ 6706.662672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6708.414830] LustreError: 237784:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup
[ 6708.417651] LustreError: 237784:0:(obd_class.h:479:obd_check_dev()) Skipped 51 previous similar messages
[ 6708.446604] Lustre: server umount lustre-MDT0000 complete
[ 6709.623780] LustreError: 219920:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767326606 with bad export cookie 15296755691748665849
[ 6709.625945] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6709.627231] LustreError: 219920:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 6709.734660] Lustre: server umount lustre-MDT0001 complete
[ 6721.538444] Lustre: server umount lustre-OST0000 complete
[ 6732.952853] Lustre: server umount lustre-OST0001 complete
[ 6741.657504] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local
[ 6747.126039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 6747.260800] LustreError: 240720:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6747.265946] LustreError: 240720:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages
[ 6747.278781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6747.280754] Lustre: Skipped 3 previous similar messages
[ 6747.283739] LustreError: lustre-MDT0000: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[ 6748.464869] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6750.210994] Lustre: 241276:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6750.215483] Lustre: 241276:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 1 previous similar message
[ 6753.448984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 6753.530801] LustreError: lustre-MDT0001: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[ 6754.717138] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6759.672355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 6759.760163] LustreError: lustre-OST0000: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[ 6761.607944] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6765.482436] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 6765.531251] LustreError: lustre-OST0001: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[ 6767.351076] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6772.897364] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:20 to 0x280000401:65)
[ 6772.897922] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:20 to 0x2c0000401:65)
[ 6775.802704] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6784.086887] LustreError: 240714:0:(osd_handler.c:3458:osd_quota_transfer()) lustre-MDT0000: quota transfer failed. Is project enforcement enabled on the ldiskfs filesystem? rc = -95
[ 6788.577874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6788.581232] Lustre: Skipped 7 previous similar messages
[ 6791.438729] Lustre: server umount lustre-MDT0000 complete
[ 6792.651343] LustreError: 240702:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767326689 with bad export cookie 15296755691748714534
[ 6792.654729] LustreError: 240702:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 6792.655177] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6792.764472] Lustre: server umount lustre-MDT0001 complete
[ 6805.055709] Lustre: server umount lustre-OST0000 complete
[ 6816.468200] Lustre: server umount lustre-OST0001 complete
[ 6825.335945] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local
[ 6830.551390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 6832.033855] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6837.343620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 6838.827605] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6844.147528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 6846.130476] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6850.038561] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 6851.990704] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6855.713543] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:20 to 0x280000401:97)
[ 6855.713788] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97)
[ 6860.430097] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6879.698354] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 23:06:16 (1767326776)
[ 6900.210543] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[ 6900.821183] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 23:06:37 (1767326797)
[ 6907.963932] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[ 6908.531947] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 23:06:45 (1767326805)
[ 6926.388432] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[ 6926.872813] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 23:07:03 (1767326823)
[ 6942.024257] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 23:07:18 (1767326838)
[ 6945.352366] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[ 6946.813241] Lustre: 258411:0:(qsd_reint.c:241:qsd_reint_index()) lustre-OST0000: index version for fid [0x200000005:0x1007:0x0] is 0, but index isn't empty (1)
[ 6948.095799] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 6949.638943] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 6950.767518] Lustre: DEBUG MARKER: Write...
[ 6958.513265] LustreError: 259623:0:(mgs_handler.c:1067:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[ 6962.158077] Lustre: DEBUG MARKER: Write...
[ 6965.027144] Lustre: DEBUG MARKER: Write...
[ 7003.303263] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 23:08:20 (1767326900)
[ 7012.828465] LustreError: 263678:0:(mgs_handler.c:1067:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[ 7033.977307] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 23:08:50 (1767326930)
[ 7044.695787] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[ 7045.263368] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[ 7074.877419] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 23:09:31 (1767326971)
[ 7091.494042] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 23:09:48 (1767326988)
[ 7108.473284] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 23:10:05 (1767327005)
[ 7111.889552] Lustre: DEBUG MARKER: User quota (block hardlimit:100 MB)
[ 7122.873389] Lustre: DEBUG MARKER: Write...
[ 7123.526959] Lustre: DEBUG MARKER: Write out of block quota ...
[ 7171.158284] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 23:11:08 (1767327068)
[ 7174.658772] Lustre: DEBUG MARKER: User quota (block hardlimit:1000 MB)
[ 7212.751378] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 23:11:49 (1767327109)
[ 7216.527929] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB)
[ 7223.330098] Lustre: DEBUG MARKER: Write...
[ 7223.978687] Lustre: DEBUG MARKER: Write out of block quota ...
[ 7224.079189] LustreError: 258417:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:10240 soft:0 granted:10240 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[ 7248.698652] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 23:12:25 (1767327145)
[ 7259.048973] Lustre: DEBUG MARKER: set to use default quota
[ 7259.628193] Lustre: DEBUG MARKER: set default quota
[ 7260.205137] Lustre: DEBUG MARKER: get default quota
[ 7261.671296] Lustre: DEBUG MARKER: Test not out of quota
[ 7262.712812] Lustre: DEBUG MARKER: Test out of quota
[ 7265.547871] Lustre: DEBUG MARKER: Increase default quota
[ 7274.154138] Lustre: DEBUG MARKER: Set quota to override default quota
[ 7274.172882] LustreError: 247924:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45056 time:1767931971 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[ 7277.253567] Lustre: DEBUG MARKER: Set to use default quota again
[ 7286.407370] Lustre: DEBUG MARKER: Cleanup
[ 7313.438052] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 23:13:30 (1767327210)
[ 7320.926915] Lustre: DEBUG MARKER: set default quota for qpool1
[ 7321.486456] Lustre: DEBUG MARKER: Write from user that hasn't lqe
[ 7346.376745] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 23:14:03 (1767327243)
[ 7378.471551] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 23:14:35 (1767327275)
[ 7401.181737] Lustre: DEBUG MARKER: Write...
[ 7402.188507] Lustre: DEBUG MARKER: Write out of block quota ...
[ 7448.029614] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 23:15:44 (1767327344)
[ 7463.216731] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 23:16:00 (1767327360)
[ 7465.476694] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 23:16:02 (1767327362)
[ 7484.162511] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 23:16:21 (1767327381)
[ 7504.470378] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 23:16:41 (1767327401)
[ 7514.157326] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 23:16:51 (1767327411)
[ 7525.989877] Lustre: *** cfs_fail_loc=a06, val=0***
[ 7525.991101] Lustre: Skipped 1127 previous similar messages
[ 7526.058756] LustreError: 247942:0:(qmt_lock.c:446:qmt_lvbo_update()) $$$ failed to release quota space on glimpse 0!=2048 : rc = -11
                 qmt:lustre-QMT0000 pool:dt-0x0 id:60000 enforced:1 hard:102400 soft:0 granted:23552 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no
[ 7529.759261] Lustre: Failing over lustre-OST0001
[ 7529.781640] LustreError: 299429:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup
[ 7529.783210] LustreError: 299429:0:(obd_class.h:479:obd_check_dev()) Skipped 51 previous similar messages
[ 7529.804269] Lustre: server umount lustre-OST0001 complete
[ 7530.300435] LustreError: 251577:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 7530.306467] LustreError: 251577:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages
[ 7531.493814] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107
[ 7531.499168] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 7531.505313] Lustre: Skipped 7 previous similar messages
[ 7534.062691] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 7534.159871] Lustre: lustre-OST0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180
[ 7534.167280] Lustre: lustre-OST0001: in recovery but waiting for the first client to connect
[ 7534.169034] Lustre: Skipped 1 previous similar message
[ 7535.415063] Lustre: lustre-OST0001: Will be in recovery for at least 1:00, or until 3 clients reconnect
[ 7535.417579] Lustre: Skipped 1 previous similar message
[ 7535.984164] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7536.109770] Lustre: lustre-OST0001: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted.
[ 7536.109774] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo)
[ 7536.109781] Lustre: Skipped 7 previous similar messages
[ 7536.113084] Lustre: Skipped 1 previous similar message
[ 7560.900619] Lustre: DEBUG MARKER: == sanity-quota test 81: Race qmt_start_pool_recalc with qmt_pool_free ========================================================== 23:17:37 (1767327457)
[ 7564.519737] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[ 7568.938745] LustreError: 302515:0:(qmt_pool.c:1295:qmt_pool_recalc()) cfs_fail_timeout id a07 sleeping for 10000ms
[ 7571.259231] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.40@tcp (stopping)
[ 7575.008810] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 7576.374269] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.40@tcp (stopping)
[ 7576.376627] Lustre: Skipped 4 previous similar messages
[ 7579.031113] LustreError: 302515:0:(qmt_pool.c:1295:qmt_pool_recalc()) cfs_fail_timeout id a07 awake
[ 7579.112413] Lustre: server umount lustre-MDT0000 complete
[ 7583.149336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 7583.187601] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 7583.273048] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 7583.275942] Lustre: Skipped 7 previous similar messages
[ 7583.303216] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:112 to 0x280000401:129)
[ 7583.306093] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:113 to 0x2c0000401:129)
[ 7584.649106] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7596.687760] Lustre: DEBUG MARKER: == sanity-quota test 82: verify more than 8 qids for single operation ========================================================== 23:18:13 (1767327493)
[ 7604.455190] Lustre: DEBUG MARKER: == sanity-quota test 83: Setting default quota shouldn't affect grace time ========================================================== 23:18:21 (1767327501)
[ 7611.720937] Lustre: DEBUG MARKER: == sanity-quota test 84: Reset quota should fix the insane granted quota ========================================================== 23:18:28 (1767327508)
[ 7630.708173] Lustre: *** cfs_fail_loc=a08, val=0***
[ 7630.709766] Lustre: Skipped 9 previous similar messages
[ 7630.712285] Lustre: *** cfs_fail_loc=a08, val=0***
[ 7630.714510] Lustre: Skipped 1 previous similar message
[ 7669.207230] Lustre: DEBUG MARKER: == sanity-quota test 85: do not hung at write with the least_qunit ========================================================== 23:19:26 (1767327566)
[ 7683.347091] LustreError: 253051:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:3072 soft:0 granted:3072 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[ 7710.365988] Lustre: DEBUG MARKER: == sanity-quota test 86: Pre-acquired quota should be released if quota is over limit ========================================================== 23:20:07 (1767327607)
[ 7726.480112] LustreError: 247924:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:16384 time:1767932423 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[ 7769.362965] LustreError: 251576:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:16384 time:1767932466 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[ 7816.722344] LustreError: 247924:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:1000 enforced:1 hard:2048 soft:2048 granted:16385 time:1767932513 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[ 7857.819936] Lustre: DEBUG MARKER: == sanity-quota test 87: lfs quota -a should print default quota setting ========================================================== 23:22:34 (1767327754)
[ 7859.775489] Lustre: *** cfs_fail_loc=a09, val=0***
[ 7859.778066] Lustre: Skipped 1 previous similar message
[ 7880.161828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 7880.162400] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 7880.164071] Lustre: Skipped 3 previous similar messages
[ 7885.092656] Lustre: server umount lustre-MDT0000 complete
[ 7888.936448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 7888.975491] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 7889.087496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 7889.131350] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:132 to 0x280000401:161)
[ 7889.131956] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:113 to 0x2c0000401:161)
[ 7890.431204] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7894.499197] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7894.511477] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo)
[ 7894.513605] Lustre: Skipped 5 previous similar messages
[ 7897.546333] Lustre: DEBUG MARKER: == sanity-quota test 88: Writing over quota should not hang ========================================================== 23:23:14 (1767327794)
[ 7898.042407] Lustre: DEBUG MARKER: SKIP: sanity-quota test_88 require client with >4k pages
[ 7898.617079] Lustre: DEBUG MARKER: == sanity-quota test 89: Show default quota with squash_uid ========================================================== 23:23:15 (1767327795)
[ 7905.263533] Lustre: DEBUG MARKER: == sanity-quota test 90a: lfs quota should work without mount point ========================================================== 23:23:22 (1767327802)
[ 7915.503675] Lustre: DEBUG MARKER: == sanity-quota test 90b: lfs quota should work with multiple mount points ========================================================== 23:23:32 (1767327812)
[ 7926.676176] Lustre: DEBUG MARKER: == sanity-quota test 91: new quota index files in quota_master ========================================================== 23:23:43 (1767327823)
[ 7940.577873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 7940.577954] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 7940.579452] Lustre: Skipped 4 previous similar messages
[ 7945.435222] Lustre: server umount lustre-MDT0000 complete
[ 7948.210196] LustreError: 247908:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767327845 with bad export cookie 15296755691749819848
[ 7948.212581] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 7948.212941] LustreError: 247908:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 7954.408809] Lustre: server umount lustre-MDT0001 complete
[ 7972.319103] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck?
[ 7972.351668] Lustre: server umount lustre-OST0000 complete
[ 7981.132198] Lustre: server umount lustre-OST0001 complete
[ 7985.574110] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_hostid
[ 7987.828490] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local
[ 7995.089204] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[ 7995.427414] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[ 7996.733547] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[ 7997.481790] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[ 8001.053103] loop: module loaded
[ 8002.530676] Lustre: DEBUG MARKER: sanity-quota test_91: @@@@@@ FAIL: Cannot start mds1
[ 8005.490295] Lustre: DEBUG MARKER: == sanity-quota test 92: Cannot set inode limit with Quota Pools ========================================================== 23:25:02 (1767327902)
[ 8006.336686] Lustre: DEBUG MARKER: sanity-quota test_92: @@@@@@ FAIL: No pool created, result code 19
[ 8007.980354] Lustre: DEBUG MARKER: sanity-quota test_92: @@@@@@ FAIL: pool_add failed
[ 8009.685776] Lustre: DEBUG MARKER: == sanity-quota test 93: update projid while client write to OST ========================================================== 23:25:06 (1767327906)
[ 8011.456231] Lustre: DEBUG MARKER: sanity-quota test_93: @@@@@@ FAIL: setup quota failed with 1
[ 8013.154705] Lustre: DEBUG MARKER: == sanity-quota test 94: lfs quota all respects nodemap offset ========================================================== 23:25:10 (1767327910)
[ 8013.739553] Lustre: DEBUG MARKER: sanity-quota test_94: @@@@@@ FAIL: Set quota for usr id 100000 failed
[ 8015.380270] Lustre: DEBUG MARKER: == sanity-quota test 95a: Correct limits for squashed root ========================================================== 23:25:12 (1767327912)
[ 8015.970804] Lustre: DEBUG MARKER: sanity-quota test_95a: @@@@@@ FAIL: Can't setquota for uid 100000