[ 2701.965870] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 2704.197130] Lustre: DEBUG MARKER: Files for user (quota_usr), count=0:
[ 2704.824915] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:10240).
[ 2705.515673] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 2708.552163] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2709.093845] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 2709.774228] Lustre: Failing over lustre-MDT0000
[ 2710.055092] Lustre: server umount lustre-MDT0000 complete
[ 2710.285325] LustreError: lustre-MDT0000-lwp-OST0000: operation quota_acquire to node 0@lo failed: rc = -107
[ 2710.288649] LustreError: 37453:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2710.296141] LustreError: 37453:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages
[ 2723.979787] LDISKFS-fs (dm-0): 7 truncates cleaned up
[ 2723.981098] LDISKFS-fs (dm-0): recovery complete
[ 2723.984575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2724.018312] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2724.092523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2724.111248] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 2725.009335] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 2725.336247] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2729.455042] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted.
[ 2729.473517] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2057 to 0x280000401:2081)
[ 2729.473623] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2021 to 0x2c0000401:2049)
[ 2731.075811] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2731.540664] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2740.120774] Lustre: DEBUG MARKER: (dd_pid=125569, time=6, timeout=600)
[ 2752.489248] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 2754.531751] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 2757.059522] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2757.521734] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 2758.102113] Lustre: Failing over lustre-MDT0000
[ 2758.410827] Lustre: server umount lustre-MDT0000 complete
[ 2760.159562] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 2760.159977] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2760.162864] LustreError: 124103:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2760.167478] Lustre: Skipped 6 previous similar messages
[ 2760.173966] LustreError: 124103:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages
[ 2772.083981] LDISKFS-fs (dm-0): 2 truncates cleaned up
[ 2772.085220] LDISKFS-fs (dm-0): recovery complete
[ 2772.087465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2772.119037] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2772.187715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2773.416601] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2777.574545] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo)
[ 2777.577687] Lustre: Skipped 5 previous similar messages
[ 2777.597254] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2083 to 0x280000401:2113)
[ 2777.597262] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2021 to 0x2c0000401:2081)
[ 2779.284124] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2779.769573] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2782.020545] Lustre: DEBUG MARKER: (dd_pid=128044, time=0, timeout=600)
[ 2801.183983] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 04:01:11 (1789891271)
[ 2823.473339] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 04:01:33 (1789891293)
[ 2832.089889] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 2835.467996] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 2835.957282] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 2836.440033] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 2837.028984] Lustre: DEBUG MARKER: Set quota for 1 times
[ 2838.634902] Lustre: DEBUG MARKER: Set quota for 2 times
[ 2840.254427] Lustre: DEBUG MARKER: Set quota for 3 times
[ 2841.832596] Lustre: DEBUG MARKER: Set quota for 4 times
[ 2843.407432] Lustre: DEBUG MARKER: Set quota for 5 times
[ 2844.967719] Lustre: DEBUG MARKER: Set quota for 6 times
[ 2846.561027] Lustre: DEBUG MARKER: Set quota for 7 times
[ 2848.083356] Lustre: DEBUG MARKER: Set quota for 8 times
[ 2849.637888] Lustre: DEBUG MARKER: Set quota for 9 times
[ 2851.167861] Lustre: DEBUG MARKER: Set quota for 10 times
[ 2852.726610] Lustre: DEBUG MARKER: Set quota for 11 times
[ 2854.294574] Lustre: DEBUG MARKER: Set quota for 12 times
[ 2855.895404] Lustre: DEBUG MARKER: Set quota for 13 times
[ 2857.427595] Lustre: DEBUG MARKER: Set quota for 14 times
[ 2859.042362] Lustre: DEBUG MARKER: Set quota for 15 times
[ 2860.658951] Lustre: DEBUG MARKER: Set quota for 16 times
[ 2862.320980] Lustre: DEBUG MARKER: Set quota for 17 times
[ 2863.868824] Lustre: DEBUG MARKER: Set quota for 18 times
[ 2865.414563] Lustre: DEBUG MARKER: Set quota for 19 times
[ 2888.839801] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 04:02:38 (1789891358)
[ 2895.328109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 2895.328319] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 2895.330134] Lustre: Skipped 3 previous similar messages
[ 2897.231146] Lustre: server umount lustre-MDT0000 complete
[ 2898.341713] LustreError: 23310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789891368 with bad export cookie 1781279686324966959
[ 2898.342738] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2898.346066] LustreError: 23310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message
[ 2898.444722] Lustre: server umount lustre-MDT0001 complete
[ 2909.693842] Lustre: server umount lustre-OST0000 complete
[ 2921.131350] Lustre: server umount lustre-OST0001 complete
[ 2925.938504] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local
[ 2929.630259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2929.778683] LustreError: 155079:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2929.788189] LustreError: 155079:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages
[ 2929.802947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2931.062721] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2933.597393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2933.706330] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 2934.904324] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2935.772159] Lustre: 156233:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 2937.705471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 2937.799157] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 2939.470418] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2939.873567] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2116 to 0x280000401:2145)
[ 2941.982554] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 2943.726309] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2944.098515] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2083 to 0x2c0000401:2113)
[ 2944.099205] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:97)
[ 2946.568044] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 2947.596961] Lustre: 158107:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 2954.207643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 2955.921648] Lustre: server umount lustre-MDT0000 complete
[ 2957.017780] LustreError: 156235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789891427 with bad export cookie 1781279686324977550
[ 2957.019228] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2957.021484] LustreError: 156235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 2959.126475] Lustre: server umount lustre-MDT0001 complete
[ 2969.534084] Lustre: server umount lustre-OST0000 complete
[ 2980.944362] Lustre: server umount lustre-OST0001 complete
[ 2986.175410] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local
[ 2989.816321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2989.978267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2989.981450] Lustre: Skipped 1 previous similar message
[ 2991.391822] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2994.064478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2995.401604] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2996.321087] Lustre: 161889:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 2998.308258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 2999.458561] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2116 to 0x280000401:2177)
[ 3000.227926] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3002.964521] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 3004.066370] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2083 to 0x2c0000401:2145)
[ 3004.067239] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:129)
[ 3004.851237] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3007.851110] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 3008.978394] Lustre: 163758:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 3011.986918] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 04:04:42 (1789891482)
[ 3012.437874] Lustre: DEBUG MARKER: OST0_SIZE: 3605396 required: 6144
[ 3012.903311] Lustre: DEBUG MARKER: run for 4MB test file
[ 3017.526221] Lustre: DEBUG MARKER: User quota (limit: 4 MB)
[ 3019.156019] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT
[ 3019.610554] Lustre: DEBUG MARKER: Write half of file
[ 3020.112428] Lustre: DEBUG MARKER: Write out of block quota ...
[ 3020.612985] Lustre: DEBUG MARKER: Step1: done
[ 3021.122602] Lustre: DEBUG MARKER: Step2: rewrite should succeed
[ 3021.604930] Lustre: DEBUG MARKER: Step2: done
[ 3035.322849] Lustre: DEBUG MARKER: OST0_SIZE: 3605396 required: 61440
[ 3035.800262] Lustre: DEBUG MARKER: run for 40MB test file
[ 3039.214088] Lustre: DEBUG MARKER: User quota (limit: 40 MB)
[ 3040.945171] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT
[ 3041.396055] Lustre: DEBUG MARKER: Write half of file
[ 3042.180755] Lustre: DEBUG MARKER: Write out of block quota ...
[ 3043.019307] Lustre: DEBUG MARKER: Step1: done
[ 3043.527993] Lustre: DEBUG MARKER: Step2: rewrite should succeed
[ 3044.001041] Lustre: DEBUG MARKER: Step2: done
[ 3064.255897] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 04:05:34 (1789891534)
[ 3085.602429] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 04:05:55 (1789891555)
[ 3096.101475] Lustre: DEBUG MARKER: Write...
[ 3096.836095] Lustre: DEBUG MARKER: Write out of block quota ...
[ 3126.807146] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 04:06:36 (1789891596)
[ 3128.842584] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 04:06:38 (1789891598)
[ 3133.139740] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 04:06:43 (1789891603)
[ 3136.592422] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 04:06:46 (1789891606)
[ 3139.665538] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 04:06:49 (1789891609)
[ 3167.109746] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 3187.440544] Lustre: DEBUG MARKER: sanity-quota test_33: @@@@@@ FAIL: Used space for user 60000:0, expected:20480
[ 3202.792769] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 3222.187637] Lustre: DEBUG MARKER: sanity-quota test_34: @@@@@@ FAIL: Used space for user 60000 is 0, expected 2048
[ 3239.258048] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 04:08:29 (1789891709)
[ 3255.041984] Lustre: DEBUG MARKER: sanity-quota test_35: @@@@@@ FAIL: Used space for user 60000 is 0, expected 2M