[ 5183.309627] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5189.411542] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 5196.781467] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5198.314078] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5200.211131] Lustre: Failing over lustre-MDT0000
[ 5200.320447] LustreError: lustre-MDT0000-lwp-OST0000: operation quota_acquire to node 0@lo failed: rc = -107
[ 5200.324692] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5200.330463] Lustre: Skipped 1 previous similar message
[ 5200.333581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5200.498027] Lustre: server umount lustre-MDT0000 complete
[ 5204.426616] LustreError: 6497:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5204.445966] LustreError: 6497:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages
[ 5221.336783] Lustre: 3626:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786808253/real 1786808253]  req@ffff8ef209789500 x1873598853958272/t0(0) o400->MGC192.168.204.129@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786808269 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5221.354625] Lustre: 3626:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 5221.358785] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5224.838355] LDISKFS-fs (dm-0): 7 truncates cleaned up
[ 5224.840456] LDISKFS-fs (dm-0): recovery complete
[ 5224.894523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5232.143864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5232.231754] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5232.383225] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 5237.247642] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 5237.262518] Lustre: Skipped 1 previous similar message
[ 5237.347303] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted.
[ 5237.400498] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:661 to 0x280000401:705)
[ 5237.400500] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:593 to 0x2c0000401:609)
[ 5239.419149] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5251.521778] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5253.125601] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5259.323283] Lustre: DEBUG MARKER: (dd_pid=128222, time=0, timeout=600)
[ 5299.201923] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5306.463423] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 5315.701932] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5317.756961] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5320.216053] Lustre: Failing over lustre-MDT0000
[ 5322.730893] Lustre: server umount lustre-MDT0000 complete
[ 5324.258938] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5324.294763] Lustre: Skipped 3 previous similar messages
[ 5324.306831] LustreError: 43341:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5324.333229] LustreError: 43341:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 36 previous similar messages
[ 5340.646716] Lustre: 3624:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786808372/real 1786808372]  req@ffff8ef32e24f800 x1873598854034432/t0(0) o400->MGC192.168.204.129@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786808388 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5340.696505] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5347.686558] LDISKFS-fs (dm-0): 3 truncates cleaned up
[ 5347.688102] LDISKFS-fs (dm-0): recovery complete
[ 5347.693192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5350.896957] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x50ec795378477772
[ 5350.915564] Lustre: MGC192.168.204.129@tcp: Connection restored to 0@lo (at 0@lo)
[ 5350.933341] Lustre: Skipped 3 previous similar messages
[ 5351.286134] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5351.343615] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5352.793486] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 5355.869767] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5356.654254] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted.
[ 5356.731628] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:593 to 0x2c0000401:641)
[ 5356.735267] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:707 to 0x280000401:737)
[ 5365.074799] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5367.559664] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5373.166353] Lustre: DEBUG MARKER: (dd_pid=130793, time=0, timeout=600)
[ 5416.616715] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 11:41:01 (1786808461)
[ 5467.183931] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 11:41:51 (1786808511)
[ 5493.909576] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 5505.448570] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 5507.197314] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 5508.774515] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 5511.141897] Lustre: DEBUG MARKER: Set quota for 1 times
[ 5514.898394] Lustre: DEBUG MARKER: Set quota for 2 times
[ 5517.851698] Lustre: DEBUG MARKER: Set quota for 3 times
[ 5521.343975] Lustre: DEBUG MARKER: Set quota for 4 times
[ 5524.829378] Lustre: DEBUG MARKER: Set quota for 5 times
[ 5528.025039] Lustre: DEBUG MARKER: Set quota for 6 times
[ 5531.136179] Lustre: DEBUG MARKER: Set quota for 7 times
[ 5533.994562] Lustre: DEBUG MARKER: Set quota for 8 times
[ 5537.262351] Lustre: DEBUG MARKER: Set quota for 9 times
[ 5586.764134] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 11:43:51 (1786808631)
[ 5607.392780] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5607.394730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5607.425992] Lustre: Skipped 2 previous similar messages
[ 5607.466463] Lustre: Skipped 6 previous similar messages
[ 5610.314385] Lustre: server umount lustre-MDT0000 complete
[ 5612.513674] LustreError: 15709:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5612.542663] LustreError: 15709:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 34 previous similar messages
[ 5614.079513] LustreError: 6483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786808661 with bad export cookie 5831169016945145714
[ 5614.084292] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5614.093743] LustreError: 6483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages
[ 5614.393491] Lustre: server umount lustre-MDT0001 complete
[ 5627.398433] Lustre: server umount lustre-OST0000 complete
[ 5640.333341] Lustre: server umount lustre-OST0001 complete
[ 5658.228668] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local
[ 5671.071895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5671.557262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5675.998498] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5684.938422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5685.362617] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 5689.646766] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5692.895759] Lustre: 154909:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 5699.529296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 5699.887514] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 5705.741534] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5710.137699] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97)
[ 5714.838514] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 5715.107682] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 5716.153200] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:739 to 0x280000401:769)
[ 5716.154579] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:644 to 0x2c0000401:673)
[ 5722.093292] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5730.378406] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 5734.113299] Lustre: 156788:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 5751.778208] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 5751.786826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5757.409668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5757.422099] Lustre: Skipped 7 previous similar messages
[ 5757.786509] Lustre: server umount lustre-MDT0000 complete
[ 5761.281892] LustreError: 153738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786808809 with bad export cookie 5831169016945153330
[ 5761.285129] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5761.296896] LustreError: 153738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 5761.561432] Lustre: server umount lustre-MDT0001 complete
[ 5775.075504] Lustre: server umount lustre-OST0000 complete
[ 5788.707702] Lustre: server umount lustre-OST0001 complete
[ 5804.038328] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local
[ 5814.398943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5814.911190] LustreError: 159384:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5814.938603] LustreError: 159384:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages
[ 5815.041588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5820.015676] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5830.316411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5834.735421] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5837.697449] Lustre: 160525:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 5845.597404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 5852.451541] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5855.338535] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:129)
[ 5859.443202] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:739 to 0x280000401:801)
[ 5863.522254] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 5863.760369] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 5863.764702] Lustre: Skipped 2 previous similar messages
[ 5869.062441] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:644 to 0x2c0000401:705)
[ 5871.244715] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5879.773586] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 5883.270992] Lustre: 162396:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 5892.995978] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 11:48:58 (1786808938)
[ 5894.348623] Lustre: DEBUG MARKER: OST0_SIZE: 3605396 required: 6144
[ 5895.474568] Lustre: DEBUG MARKER: run for 4MB test file
[ 5904.930203] Lustre: DEBUG MARKER: User quota (limit: 4 MB)
[ 5910.620039] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT
[ 5911.884139] Lustre: DEBUG MARKER: Write half of file
[ 5913.277955] Lustre: DEBUG MARKER: Write out of block quota ...
[ 5915.138641] Lustre: DEBUG MARKER: Step1: done
[ 5916.547489] Lustre: DEBUG MARKER: Step2: rewrite should succeed
[ 5917.930503] Lustre: DEBUG MARKER: Step2: done
[ 5942.348405] Lustre: DEBUG MARKER: OST0_SIZE: 3605396 required: 61440
[ 5944.936732] Lustre: DEBUG MARKER: run for 40MB test file
[ 5958.650520] Lustre: DEBUG MARKER: User quota (limit: 40 MB)
[ 5966.524950] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT
[ 5968.897799] Lustre: DEBUG MARKER: Write half of file
[ 5973.192703] Lustre: DEBUG MARKER: Write out of block quota ...
[ 5977.060594] Lustre: DEBUG MARKER: Step1: done
[ 5979.091719] Lustre: DEBUG MARKER: Step2: rewrite should succeed
[ 5981.418871] Lustre: DEBUG MARKER: Step2: done
[ 6029.920220] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 11:51:14 (1786809074)
[ 6073.027641] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 11:51:57 (1786809117)
[ 6110.208519] Lustre: DEBUG MARKER: Write...
[ 6112.579761] Lustre: DEBUG MARKER: Write out of block quota ...
[ 6171.002252] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 11:53:36 (1786809216)
[ 6177.219845] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 11:53:42 (1786809222)
[ 6186.957868] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 11:53:52 (1786809232)
[ 6194.800851] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 11:53:59 (1786809239)
[ 6203.150163] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 11:54:08 (1786809248)
[ 6254.724741] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 6381.743217] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 6517.387327] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 11:59:22 (1786809562)
[ 6564.483163] Lustre: DEBUG MARKER: Restart...
[ 6568.417507] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6568.418573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6568.447268] Lustre: Skipped 6 previous similar messages
[ 6573.538767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6573.549824] Lustre: Skipped 6 previous similar messages
[ 6574.485989] Lustre: server umount lustre-MDT0000 complete
[ 6578.128836] LustreError: 167526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786809626 with bad export cookie 5831169016945155633
[ 6578.131363] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6578.141312] LustreError: 167526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 6578.699958] Lustre: server umount lustre-MDT0001 complete
[ 6593.179739] Lustre: server umount lustre-OST0000 complete
[ 6595.039167] Lustre: 3626:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786809626/real 1786809626]  req@ffff8ef33f3d7480 x1873598854786560/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786809642 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6596.751874] Lustre: server umount lustre-OST0001 complete
[ 6610.913145] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local
[ 6622.394582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 6622.909572] LustreError: 185373:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6622.948177] LustreError: 185373:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages
[ 6623.117938] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6627.609440] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6635.871917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 6636.339924] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 6640.341089] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6642.810180] Lustre: 186517:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6649.378370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 6656.588276] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6662.087816] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:161)
[ 6667.196600] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:813 to 0x280000401:833)
[ 6667.874179] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 6668.508147] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6668.515521] Lustre: Skipped 1 previous similar message
[ 6673.919936] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:713 to 0x2c0000401:737)
[ 6676.543208] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6684.449463] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6688.982388] Lustre: 188387:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6761.460696] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 12:03:26 (1786809806)
[ 6817.425326] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 12:04:22 (1786809862)
[ 8594.676621] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 12:33:59 (1786811639)
[ 8607.725936] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 8607.737931] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 8607.759440] Lustre: Skipped 3 previous similar messages
[ 8607.768649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 8609.252531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 8612.386513] Lustre: server umount lustre-MDT0000 complete
[ 8614.372729] LustreError: 185369:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 8614.394672] LustreError: 185369:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages
[ 8616.572815] LustreError: 192575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786811664 with bad export cookie 5831169016945164733
[ 8616.574123] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 8616.586381] LustreError: 192575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 8617.046027] Lustre: server umount lustre-MDT0001 complete
[ 8632.390600] Lustre: server umount lustre-OST0000 complete
[ 8635.626268] Lustre: 3625:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786811667/real 1786811667]  req@ffff8ef20c950700 x1873598858317952/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786811683 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 8635.658339] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 8635.666644] Lustre: Skipped 3 previous similar messages
[ 8637.327890] Lustre: server umount lustre-OST0001 complete
[ 8654.448472] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local
[ 8667.401781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 8668.088658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 8673.006675] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8682.463611] LustreError: 195838:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 8682.477883] LustreError: 195838:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages
[ 8683.114621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 8683.549757] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 8688.644551] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8691.915178] Lustre: 196978:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 8698.913229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 8699.251365] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 8705.046561] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8708.479267] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:193)
[ 8712.577337] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5836 to 0x280000401:5857)
[ 8713.950393] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 8719.873432] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5738 to 0x2c0000401:5761)
[ 8721.395177] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8730.821558] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 8734.033573] Lustre: 198850:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 8759.811604] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 12:36:45 (1786811805)
[ 8794.721709] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 12:37:18 (1786811838)
[ 8825.240472] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 12:37:50 (1786811870)
[ 8862.744841] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 12:38:27 (1786811907)
[ 8905.884304] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 12:39:10 (1786811950)
[ 8988.821838] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 12:40:33 (1786812033)
[ 9023.388502] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 12:41:08 (1786812068)
[ 9091.896564] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 12:42:17 (1786812137)
[ 9097.662534] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9097.676556] Lustre: Skipped 2 previous similar messages
[ 9099.697462] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9099.699222] Lustre: Skipped 87 previous similar messages
[ 9103.701281] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9103.707405] Lustre: Skipped 167 previous similar messages
[ 9111.753114] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9111.757190] Lustre: Skipped 249 previous similar messages
[ 9127.789882] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9127.797398] Lustre: Skipped 633 previous similar messages
[ 9159.791300] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9159.800954] Lustre: Skipped 1059 previous similar messages
[ 9223.817113] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9223.819167] Lustre: Skipped 2103 previous similar messages
[ 9743.328541] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 9743.338104] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 9743.362680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 9743.377839] Lustre: Skipped 3 previous similar messages
[ 9743.868210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 9743.880246] Lustre: Skipped 2 previous similar messages
[ 9747.156117] Lustre: server umount lustre-MDT0000 complete
[ 9748.964851] LustreError: 195833:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9749.026187] LustreError: 195833:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages
[ 9752.620187] LustreError: 204539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786812800 with bad export cookie 5831169016946944980
[ 9752.630326] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 9754.082961] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 9754.086922] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping)
[ 9754.121642] Lustre: Skipped 4 previous similar messages
[ 9754.149670] Lustre: Skipped 1 previous similar message
[ 9757.097983] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping)
[ 9757.112619] Lustre: Skipped 1 previous similar message
[ 9759.702299] Lustre: server umount lustre-MDT0001 complete
[ 9766.238655] Lustre: server umount lustre-OST0000 complete
[ 9771.773669] Lustre: server umount lustre-OST0001 complete
[ 9780.585876] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid
[ 9789.360894] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local
[ 9844.443774] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local
[ 9856.231145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 9856.609664] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000
[ 9856.680686] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[ 9856.780698] Lustre: lustre-MDT0000: new disk, initializing
[ 9856.863544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 9856.883300] Lustre: Skipped 1 previous similar message
[ 9856.927692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[ 9861.914761] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9874.487074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 9874.682170] Lustre: 215622:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17
[ 9874.763089] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
[ 9874.771886] Lustre: Skipped 1 previous similar message
[ 9874.895799] Lustre: lustre-MDT0001: new disk, initializing
[ 9875.005581] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 9875.031788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt
[ 9875.049827] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt]
[ 9880.009117] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9885.612070] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params
[ 9892.680792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 9892.898922] Lustre: lustre-OST0000: new disk, initializing
[ 9892.902432] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[ 9892.906587] Lustre: 217259:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0]
[ 9892.960122] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 9894.890968] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost
[ 9894.908318] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost]
[ 9894.971187] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401
[ 9899.735260] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9910.380290] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 9910.513918] Lustre: lustre-OST0001: new disk, initializing
[ 9910.517985] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[ 9910.525166] Lustre: 218130:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0]
[ 9910.613209] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 9911.987076] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost
[ 9911.993359] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost]
[ 9912.043508] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401
[ 9917.273343] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9927.353966] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 9931.773199] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params
[ 9961.245881] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 12:56:45 (1786813005)
[ 9970.585804] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 12:56:55 (1786813015)
[10007.354703] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 12:57:32 (1786813052)
[10064.073086] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 12:58:29 (1786813109)
[10089.827903] Lustre: DEBUG MARKER: rename directory return 255
[10131.858630] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 12:59:37 (1786813177)
[10154.765605] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 12:59:59 (1786813199)
[10192.800671] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 13:00:37 (1786813237)
[10286.750376] LustreError: 226543:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-0x0 id:60001 enforced:1 hard:51200 soft:0 granted:51200 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[10318.488392] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 13:02:43 (1786813363)
[10346.925736] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 13:03:10 (1786813390)
[10387.048668] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 13:03:51 (1786813431)
[10524.809275] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 13:06:09 (1786813569)
[10530.272851] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[10530.275607] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[10530.310214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[10531.815265] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[10531.827622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[10531.842275] Lustre: Skipped 2 previous similar messages
[10531.861957] Lustre: Skipped 3 previous similar messages
[10535.969261] Lustre: server umount lustre-MDT0000 complete
[10536.937734] LustreError: 225196:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[10536.975543] LustreError: 225196:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages
[10540.228821] LustreError: 216888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786813588 with bad export cookie 5831169016947317149
[10540.235294] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[10540.243764] LustreError: 216888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages
[10540.845516] Lustre: server umount lustre-MDT0001 complete
[10555.853718] Lustre: server umount lustre-OST0000 complete
[10557.731097] Lustre: 3625:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786813589/real 1786813589]  req@ffff8ef20a44ad80 x1873598865453312/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786813605 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[10557.750024] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[10560.479706] Lustre: 3623:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786813592/real 1786813592]  req@ffff8ef20a449500 x1873598865453568/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786813608 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[10561.149791] Lustre: server umount lustre-OST0001 complete
[10588.268048] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local
[10601.656714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[10602.156105] LustreError: 235272:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[10602.245659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[10602.253274] LustreError: lustre-MDT0000: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[10606.813840] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10607.586583] LustreError: 235273:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[10616.509225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[10616.922202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[10616.929582] LustreError: lustre-MDT0001: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[10622.520747] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10626.085778] Lustre: 236415:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[10634.902508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[10635.208189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[10635.214560] LustreError: lustre-OST0000: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[10641.281959] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10641.320619] LustreError: 236769:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[10641.350045] LustreError: 236769:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message
[10647.531582] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:20 to 0x280000401:65)
[10650.811726] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[10650.974198] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[10650.979063] LustreError: lustre-OST0001: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[10656.246854] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:20 to 0x2c0000401:65)
[10657.502476] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10664.628801] Lustre: DEBUG MARKER: Using TIMEOUT=20
[10668.229556] Lustre: 238290:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[10675.252351] LustreError: 236478:0:(osd_handler.c:3426:osd_quota_transfer()) lustre-MDT0000: quota transfer failed. Is project enforcement enabled on the ldiskfs filesystem? rc = -95
[10681.827104] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[10681.829328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[10681.854589] Lustre: Skipped 2 previous similar messages
[10681.870185] Lustre: Skipped 3 previous similar messages
[10686.139649] Lustre: server umount lustre-MDT0000 complete
[10686.945137] LustreError: 235269:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[10686.960705] LustreError: 235269:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages
[10689.495941] LustreError: 235254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786813737 with bad export cookie 5831169016947366009
[10689.510036] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[10689.516658] LustreError: 235254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages
[10689.848740] Lustre: server umount lustre-MDT0001 complete
[10704.252680] Lustre: server umount lustre-OST0000 complete
[10712.031582] LustreError: 3625:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED  req@ffff8ef33fce7b80 x1873598865509888/t0(0) o601->lustre-MDT0000-lwp-OST0001@0@lo:23/10 lens 336/336 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[10712.058757] LustreError: 3626:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8  qsd:lustre-OST0001 qtype:grp lqe: ffff8ef20a696480 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 1516 qunit:0 qtune:0 edquot:0 default:no revoke:0
[10712.075030] LustreError: 3626:0:(qsd_handler.c:298:qsd_req_completion()) Skipped 1 previous similar message
[10718.358928] Lustre: server umount lustre-OST0001 complete
[10738.011942] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local
[10750.084134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[10750.587804] LustreError: 240682:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[10750.671386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[10756.034988] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10767.490594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[10774.928529] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10779.067880] Lustre: 241830:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[10786.915271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[10787.212039] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[10787.214631] Lustre: Skipped 1 previous similar message
[10794.759923] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10795.442854] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:20 to 0x280000401:97)
[10805.298474] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[10810.888279] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97)
[10811.827673] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10820.291751] Lustre: DEBUG MARKER: Using TIMEOUT=20
[10824.179635] Lustre: 243710:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[10856.915859] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 13:11:41 (1786813901)
[10903.675371] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[10905.472869] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 13:12:30 (1786813950)
[10931.112806] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[10933.638830] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 13:12:58 (1786813978)
[10974.002384] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[10976.031853] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 13:13:41 (1786814021)
[11004.944428] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 13:14:09 (1786814049)
[11016.446678] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[11022.230532] Lustre: 250220:0:(qsd_reint.c:249:qsd_reint_index()) lustre-OST0000: index version for fid [0x200000005:0x100c:0x0] is 0, but index isn't empty (1)
[11027.073590] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[11031.930789] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[11035.471422] Lustre: DEBUG MARKER: Write...
[11052.796215] LustreError: 251566:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[11058.378975] Lustre: DEBUG MARKER: Write...
[11071.138837] Lustre: DEBUG MARKER: Write...
[11139.908608] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 13:16:24 (1786814184)
[11162.895921] LustreError: 255580:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[11207.331916] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 13:17:32 (1786814252)
[11237.012326] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[11238.607853] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[11318.433397] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 13:19:23 (1786814363)
[11361.522562] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 13:20:06 (1786814406)
[11398.086358] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 13:20:43 (1786814443)
[11409.221577] Lustre: DEBUG MARKER: User quota (block hardlimit:100 MB)
[11433.824921] Lustre: DEBUG MARKER: Write...
[11435.703215] Lustre: DEBUG MARKER: Write out of block quota ...
[11535.985880] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 13:23:00 (1786814580)
[11549.215673] Lustre: DEBUG MARKER: User quota (block hardlimit:1000 MB)
[11637.745923] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 13:24:42 (1786814682)
[11651.759337] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB)
[11674.497644] Lustre: DEBUG MARKER: Write...
[11677.061766] Lustre: DEBUG MARKER: Write out of block quota ...
[11677.538109] LustreError: 240680:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:10240 soft:0 granted:10240 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[11733.304842] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 13:26:18 (1786814778)
[11762.820858] Lustre: DEBUG MARKER: set to use default quota
[11764.415480] Lustre: DEBUG MARKER: set default quota
[11767.039801] Lustre: DEBUG MARKER: get default quota
[11774.671561] Lustre: DEBUG MARKER: Test not out of quota
[11779.941388] Lustre: DEBUG MARKER: Test out of quota
[11792.439622] Lustre: DEBUG MARKER: Increase default quota
[11816.752852] Lustre: DEBUG MARKER: Set quota to override default quota
[11817.024905] LustreError: 240678:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45056 time:1787419664 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[11829.918882] Lustre: DEBUG MARKER: Set to use default quota again
[11854.385095] Lustre: DEBUG MARKER: Cleanup
[11918.877969] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 13:29:23 (1786814963)
[11943.221942] Lustre: DEBUG MARKER: set default quota for qpool1
[11945.661515] Lustre: DEBUG MARKER: Write from user that hasn't lqe
[11990.624701] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 13:30:35 (1786815035)
[12089.112399] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 13:32:13 (1786815133)
[12162.354035] Lustre: DEBUG MARKER: Write...
[12165.120886] Lustre: DEBUG MARKER: Write out of block quota ...
[12271.418437] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 13:35:16 (1786815316)
[12309.215559] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 13:35:53 (1786815353)
[12317.792448] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 13:36:03 (1786815363)
[12359.625069] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 13:36:44 (1786815404)
[12411.845921] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 13:37:36 (1786815456)
[12440.585867] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 13:38:04 (1786815484)
[12471.597544] Lustre: *** cfs_fail_loc=a06, val=0***
[12471.600718] Lustre: Skipped 3695 previous similar messages
[12471.836025] LustreError: 240696:0:(qmt_lock.c:479:qmt_lvbo_update()) $$$ failed to release quota space on glimpse 0!=2048 : rc = -11
                 qmt:lustre-QMT0000 pool:dt-0x0 id:60000 enforced:1 hard:102400 soft:0 granted:24576 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no
[12474.048430] Lustre: DEBUG MARKER: Files for user (quota_usr), count=5:
[12476.017830] Lustre: DEBUG MARKER: sanity-quota test_80: @@@@@@ FAIL: write failed