[ 2701.965870] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 2704.197130] Lustre: DEBUG MARKER: Files for user (quota_usr), count=0:
[ 2704.824915] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:10240).
[ 2705.515673] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 2708.552163] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2709.093845] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 2709.774228] Lustre: Failing over lustre-MDT0000
[ 2710.055092] Lustre: server umount lustre-MDT0000 complete
[ 2710.285325] LustreError: lustre-MDT0000-lwp-OST0000: operation quota_acquire to node 0@lo failed: rc = -107
[ 2710.288649] LustreError: 37453:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2710.296141] LustreError: 37453:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages
[ 2723.979787] LDISKFS-fs (dm-0): 7 truncates cleaned up
[ 2723.981098] LDISKFS-fs (dm-0): recovery complete
[ 2723.984575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2724.018312] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2724.092523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2724.111248] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 2725.009335] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 2725.336247] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2729.455042] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted.
[ 2729.473517] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2057 to 0x280000401:2081)
[ 2729.473623] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2021 to 0x2c0000401:2049)
[ 2731.075811] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2731.540664] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2740.120774] Lustre: DEBUG MARKER: (dd_pid=125569, time=6, timeout=600)
[ 2752.489248] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 2754.531751] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 2757.059522] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2757.521734] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 2758.102113] Lustre: Failing over lustre-MDT0000
[ 2758.410827] Lustre: server umount lustre-MDT0000 complete
[ 2760.159562] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 2760.159977] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2760.162864] LustreError: 124103:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2760.167478] Lustre: Skipped 6 previous similar messages
[ 2760.173966] LustreError: 124103:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages
[ 2772.083981] LDISKFS-fs (dm-0): 2 truncates cleaned up
[ 2772.085220] LDISKFS-fs (dm-0): recovery complete
[ 2772.087465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2772.119037] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2772.187715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2773.416601] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2777.574545] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo)
[ 2777.577687] Lustre: Skipped 5 previous similar messages
[ 2777.597254] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2083 to 0x280000401:2113)
[ 2777.597262] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2021 to 0x2c0000401:2081)
[ 2779.284124] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2779.769573] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2782.020545] Lustre: DEBUG MARKER: (dd_pid=128044, time=0, timeout=600)
[ 2801.183983] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 04:01:11 (1789891271)
[ 2823.473339] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 04:01:33 (1789891293)
[ 2832.089889] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 2835.467996] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 2835.957282] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 2836.440033] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 2837.028984] Lustre: DEBUG MARKER: Set quota for 1 times
[ 2838.634902] Lustre: DEBUG MARKER: Set quota for 2 times
[ 2840.254427] Lustre: DEBUG MARKER: Set quota for 3 times
[ 2841.832596] Lustre: DEBUG MARKER: Set quota for 4 times
[ 2843.407432] Lustre: DEBUG MARKER: Set quota for 5 times
[ 2844.967719] Lustre: DEBUG MARKER: Set quota for 6 times
[ 2846.561027] Lustre: DEBUG MARKER: Set quota for 7 times
[ 2848.083356] Lustre: DEBUG MARKER: Set quota for 8 times
[ 2849.637888] Lustre: DEBUG MARKER: Set quota for 9 times
[ 2851.167861] Lustre: DEBUG MARKER: Set quota for 10 times
[ 2852.726610] Lustre: DEBUG MARKER: Set quota for 11 times
[ 2854.294574] Lustre: DEBUG MARKER: Set quota for 12 times
[ 2855.895404] Lustre: DEBUG MARKER: Set quota for 13 times
[ 2857.427595] Lustre: DEBUG MARKER: Set quota for 14 times
[ 2859.042362] Lustre: DEBUG MARKER: Set quota for 15 times
[ 2860.658951] Lustre: DEBUG MARKER: Set quota for 16 times
[ 2862.320980] Lustre: DEBUG MARKER: Set quota for 17 times
[ 2863.868824] Lustre: DEBUG MARKER: Set quota for 18 times
[ 2865.414563] Lustre: DEBUG MARKER: Set quota for 19 times
[ 2888.839801] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 04:02:38 (1789891358)
[ 2895.328109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 2895.328319] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 2895.330134] Lustre: Skipped 3 previous similar messages
[ 2897.231146] Lustre: server umount lustre-MDT0000 complete
[ 2898.341713] LustreError: 23310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789891368 with bad export cookie 1781279686324966959
[ 2898.342738] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2898.346066] LustreError: 23310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message
[ 2898.444722] Lustre: server umount lustre-MDT0001 complete
[ 2909.693842] Lustre: server umount lustre-OST0000 complete
[ 2921.131350] Lustre: server umount lustre-OST0001 complete
[ 2925.938504] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local
[ 2929.630259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2929.778683] LustreError: 155079:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2929.788189] LustreError: 155079:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages
[ 2929.802947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2931.062721] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2933.597393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2933.706330] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 2934.904324] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2935.772159] Lustre: 156233:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 2937.705471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 2937.799157] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 2939.470418] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2939.873567] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2116 to 0x280000401:2145)
[ 2941.982554] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 2943.726309] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2944.098515] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2083 to 0x2c0000401:2113)
[ 2944.099205] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:97)
[ 2946.568044] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 2947.596961] Lustre: 158107:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 2954.207643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 2955.921648] Lustre: server umount lustre-MDT0000 complete
[ 2957.017780] LustreError: 156235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789891427 with bad export cookie 1781279686324977550
[ 2957.019228] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2957.021484] LustreError: 156235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 2959.126475] Lustre: server umount lustre-MDT0001 complete
[ 2969.534084] Lustre: server umount lustre-OST0000 complete
[ 2980.944362] Lustre: server umount lustre-OST0001 complete
[ 2986.175410] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local
[ 2989.816321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2989.978267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2989.981450] Lustre: Skipped 1 previous similar message
[ 2991.391822] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2994.064478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2995.401604] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2996.321087] Lustre: 161889:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 2998.308258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 2999.458561] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2116 to 0x280000401:2177)
[ 3000.227926] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3002.964521] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 3004.066370] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2083 to 0x2c0000401:2145)
[ 3004.067239] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:129)
[ 3004.851237] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3007.851110] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 3008.978394] Lustre: 163758:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 3011.986918] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 04:04:42 (1789891482)
[ 3012.437874] Lustre: DEBUG MARKER: OST0_SIZE: 3605396 required: 6144
[ 3012.903311] Lustre: DEBUG MARKER: run for 4MB test file
[ 3017.526221] Lustre: DEBUG MARKER: User quota (limit: 4 MB)
[ 3019.156019] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT
[ 3019.610554] Lustre: DEBUG MARKER: Write half of file
[ 3020.112428] Lustre: DEBUG MARKER: Write out of block quota ...
[ 3020.612985] Lustre: DEBUG MARKER: Step1: done
[ 3021.122602] Lustre: DEBUG MARKER: Step2: rewrite should succeed
[ 3021.604930] Lustre: DEBUG MARKER: Step2: done
[ 3035.322849] Lustre: DEBUG MARKER: OST0_SIZE: 3605396 required: 61440
[ 3035.800262] Lustre: DEBUG MARKER: run for 40MB test file
[ 3039.214088] Lustre: DEBUG MARKER: User quota (limit: 40 MB)
[ 3040.945171] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT
[ 3041.396055] Lustre: DEBUG MARKER: Write half of file
[ 3042.180755] Lustre: DEBUG MARKER: Write out of block quota ...
[ 3043.019307] Lustre: DEBUG MARKER: Step1: done
[ 3043.527993] Lustre: DEBUG MARKER: Step2: rewrite should succeed
[ 3044.001041] Lustre: DEBUG MARKER: Step2: done
[ 3064.255897] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 04:05:34 (1789891534)
[ 3085.602429] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 04:05:55 (1789891555)
[ 3096.101475] Lustre: DEBUG MARKER: Write...
[ 3096.836095] Lustre: DEBUG MARKER: Write out of block quota ...
[ 3126.807146] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 04:06:36 (1789891596)
[ 3128.842584] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 04:06:38 (1789891598)
[ 3133.139740] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 04:06:43 (1789891603)
[ 3136.592422] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 04:06:46 (1789891606)
[ 3139.665538] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 04:06:49 (1789891609)
[ 3167.109746] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 3187.440544] Lustre: DEBUG MARKER: sanity-quota test_33: @@@@@@ FAIL: Used space for user 60000:0, expected:20480
[ 3202.792769] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 3222.187637] Lustre: DEBUG MARKER: sanity-quota test_34: @@@@@@ FAIL: Used space for user 60000 is 0, expected 2048
[ 3239.258048] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 04:08:29 (1789891709)
[ 3255.041984] Lustre: DEBUG MARKER: sanity-quota test_35: @@@@@@ FAIL: Used space for user 60000 is 0, expected 2M
[ 3269.204403] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 04:08:59 (1789891739)
[ 3283.822544] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 3284.349323] Lustre: DEBUG MARKER: sanity-quota test_37: @@@@@@ FAIL: Used space is 0
[ 3299.864085] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 04:09:29 (1789891769)
[ 3714.972756] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 04:16:24 (1789892184)
[ 3720.159337] LustreError: 3627:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED  req@ffff8fe8c6045500 x1876834187992192/t0(0) o601->lustre-MDT0000-lwp-MDT0000@0@lo:23/10 lens 336/336 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 3720.159929] LustreError: lustre-MDT0000-lwp-OST0001: operation quota_acquire to node 0@lo failed: rc = -107
[ 3720.159972] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 3720.159980] Lustre: Skipped 9 previous similar messages
[ 3720.160360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 3720.160364] Lustre: Skipped 3 previous similar messages
[ 3720.167115] LustreError: 3627:0:(qsd_handler.c:318:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8  qsd:lustre-MDT0000 qtype:usr lqe: ffff8fea010ac720 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 2572 qunit:0 qtune:0 edquot:0 default:no revoke:0
[ 3720.170084] LustreError: Skipped 2 previous similar messages
[ 3721.184978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 3721.187276] Lustre: Skipped 5 previous similar messages
[ 3724.035608] Lustre: server umount lustre-MDT0000 complete
[ 3725.203933] LustreError: 164493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789892195 with bad export cookie 1781279686324979860
[ 3725.207370] LustreError: 164493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 3725.207596] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3725.314721] Lustre: server umount lustre-MDT0001 complete
[ 3736.782717] Lustre: server umount lustre-OST0000 complete
[ 3748.331781] Lustre: server umount lustre-OST0001 complete
[ 3753.378191] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local
[ 3757.045382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3757.191231] LustreError: 188491:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3757.200323] LustreError: 188491:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages
[ 3757.221421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3757.224154] Lustre: Skipped 3 previous similar messages
[ 3758.512201] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3761.149334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3761.262929] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 3762.477185] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3763.360864] Lustre: 189634:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 3765.298691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 3765.400635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 3766.432967] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7190 to 0x280000401:7233)
[ 3767.123119] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3769.687310] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 3770.787760] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7154 to 0x2c0000401:7169)
[ 3770.788338] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:161)
[ 3771.683207] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3774.746992] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 3775.881134] Lustre: 191507:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 3790.482749] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 04:17:40 (1789892260)
[ 3805.529358] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 04:17:55 (1789892275)
[ 3821.584612] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 04:18:11 (1789892291)
[ 3825.000145] Lustre: DEBUG MARKER: sanity-quota test_40c: @@@@@@ FAIL: file count expected 3 got 2
[ 3841.276818] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 04:18:31 (1789892311)
[ 3857.517924] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 04:18:47 (1789892327)
[ 3897.969653] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 04:19:27 (1789892367)
[ 3909.289293] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 04:19:39 (1789892379)
[ 3934.066308] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 04:20:04 (1789892404)
[ 3935.944517] Lustre: *** cfs_fail_loc=a09, val=0***
[ 3936.448758] Lustre: *** cfs_fail_loc=a09, val=0***
[ 3936.450196] Lustre: Skipped 99 previous similar messages
[ 3937.454593] Lustre: *** cfs_fail_loc=a09, val=0***
[ 3937.456084] Lustre: Skipped 191 previous similar messages
[ 3939.454100] Lustre: *** cfs_fail_loc=a09, val=0***
[ 3939.455379] Lustre: Skipped 399 previous similar messages
[ 3943.464708] Lustre: *** cfs_fail_loc=a09, val=0***
[ 3943.466154] Lustre: Skipped 791 previous similar messages
[ 3951.464747] Lustre: *** cfs_fail_loc=a09, val=0***
[ 3951.466289] Lustre: Skipped 1656 previous similar messages
[ 3967.467092] Lustre: *** cfs_fail_loc=a09, val=0***
[ 3967.468294] Lustre: Skipped 3082 previous similar messages
[ 4088.288482] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 4088.290351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 4088.293482] Lustre: Skipped 5 previous similar messages
[ 4088.295366] Lustre: Skipped 1 previous similar message
[ 4093.409666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 4093.412044] Lustre: Skipped 4 previous similar messages
[ 4093.765284] Lustre: server umount lustre-MDT0000 complete
[ 4095.012086] LustreError: 188472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789892565 with bad export cookie 1781279686326724925
[ 4095.014278] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4095.015574] LustreError: 188472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 4098.528385] LustreError: 189226:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 4098.529573] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping)
[ 4098.540047] LustreError: 189226:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages
[ 4098.543898] Lustre: Skipped 1 previous similar message
[ 4109.279126] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck?
[ 4109.400837] Lustre: server umount lustre-MDT0001 complete
[ 4110.962174] Lustre: server umount lustre-OST0000 complete
[ 4112.422829] Lustre: server umount lustre-OST0001 complete
[ 4114.634890] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_hostid
[ 4117.558907] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local
[ 4133.658049] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local
[ 4137.220844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4137.304982] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000
[ 4137.316195] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[ 4137.352969] Lustre: lustre-MDT0000: new disk, initializing
[ 4137.372825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4137.374957] Lustre: Skipped 1 previous similar message
[ 4137.379461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[ 4138.668691] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4142.720750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4142.752111] Lustre: 208990:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17
[ 4142.762639] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
[ 4142.764596] Lustre: Skipped 1 previous similar message
[ 4142.798482] Lustre: lustre-MDT0001: new disk, initializing
[ 4142.825466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt
[ 4142.828569] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt]
[ 4144.107734] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4146.453751] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params
[ 4148.548120] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 4148.640193] Lustre: lustre-OST0000: new disk, initializing
[ 4148.642134] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[ 4148.644710] Lustre: 210626:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0]
[ 4149.611395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost
[ 4149.614402] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost]
[ 4149.626122] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401
[ 4150.526670] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4154.662056] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 4154.703020] Lustre: lustre-OST0001: new disk, initializing
[ 4154.704712] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[ 4154.707045] Lustre: 211495:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0]
[ 4154.724514] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 4154.728076] Lustre: Skipped 2 previous similar messages
[ 4156.459233] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost
[ 4156.462135] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost]
[ 4156.475110] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401
[ 4156.604579] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4161.052609] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 4162.224581] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params
[ 4171.693444] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 04:24:01 (1789892641)
[ 4173.706496] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 04:24:03 (1789892643)
[ 4187.242738] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 04:24:17 (1789892657)
[ 4211.762548] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 04:24:41 (1789892681)
[ 4218.605828] Lustre: DEBUG MARKER: rename directory return 255
[ 4238.130776] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 04:25:08 (1789892708)
[ 4245.195411] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 04:25:15 (1789892715)
[ 4254.434167] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 04:25:24 (1789892724)
[ 4284.355936] LustreError: 220256:0:(qmt_entry.c:559:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-0x0 id:60001 enforced:1 hard:51200 soft:0 granted:51200 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[ 4298.010994] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 04:26:07 (1789892767)
[ 4307.320330] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 04:26:17 (1789892777)
[ 4325.256823] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 04:26:35 (1789892795)
[ 4376.302219] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 04:27:26 (1789892846)
[ 4378.591627] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 4378.594768] LustreError: Skipped 2 previous similar messages
[ 4378.596463] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 4378.600444] Lustre: Skipped 3 previous similar messages
[ 4378.602053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 4378.604076] Lustre: Skipped 4 previous similar messages
[ 4383.711881] Lustre: server umount lustre-MDT0000 complete
[ 4384.932893] LustreError: 208983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789892855 with bad export cookie 1781279686327097136
[ 4384.936310] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4384.938503] LustreError: 208983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 4385.051454] Lustre: server umount lustre-MDT0001 complete
[ 4395.969654] Lustre: server umount lustre-OST0000 complete
[ 4406.581964] Lustre: server umount lustre-OST0001 complete
[ 4412.699576] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local
[ 4416.091510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4416.222066] LustreError: 229335:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 4416.227826] LustreError: 229335:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages
[ 4416.242696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4416.246431] LustreError: lustre-MDT0000: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[ 4417.587834] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4420.391544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4420.509696] LustreError: lustre-MDT0001: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[ 4421.793913] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4422.788858] Lustre: 230478:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 4424.900853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 4424.997175] LustreError: lustre-OST0000: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[ 4426.018564] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:20 to 0x280000401:65)
[ 4426.786038] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4429.561023] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 4429.611349] LustreError: lustre-OST0001: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[ 4430.625964] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:20 to 0x2c0000401:65)
[ 4431.554691] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4434.588573] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 4435.746960] Lustre: 232349:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 4437.905487] LustreError: 229331:0:(osd_handler.c:3426:osd_quota_transfer()) lustre-MDT0000: quota transfer failed. Is project enforcement enabled on the ldiskfs filesystem? rc = -95
[ 4441.058512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 4441.059242] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 4441.060361] Lustre: Skipped 7 previous similar messages
[ 4445.266592] Lustre: server umount lustre-MDT0000 complete
[ 4446.516983] LustreError: 229317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789892917 with bad export cookie 1781279686327145709
[ 4446.520245] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4446.521279] LustreError: 229317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 4446.631599] Lustre: server umount lustre-MDT0001 complete
[ 4457.473110] Lustre: server umount lustre-OST0000 complete
[ 4469.006946] Lustre: server umount lustre-OST0001 complete
[ 4475.250886] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local
[ 4478.534863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4479.947102] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4482.698490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4482.792989] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 4482.795993] Lustre: Skipped 4 previous similar messages
[ 4484.097964] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4485.032920] Lustre: 235882:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 4487.163105] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 4489.118353] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4489.313692] LustreError: 236235:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 4489.315827] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97)
[ 4489.320713] LustreError: 236235:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages
[ 4491.900562] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 4492.962133] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:20 to 0x2c0000401:97)
[ 4493.833126] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4496.840886] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 4513.463508] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 04:29:43 (1789892983)
[ 4533.593447] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[ 4534.106138] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 04:30:04 (1789893004)
[ 4540.966348] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[ 4541.550659] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 04:30:11 (1789893011)
[ 4546.595711] Lustre: DEBUG MARKER: sanity-quota test_64: @@@@@@ FAIL: expected 0 got 6
[ 4559.844357] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[ 4560.334855] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 04:30:30 (1789893030)
[ 4580.035077] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 04:30:50 (1789893050)
[ 4583.629248] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[ 4585.484981] Lustre: 245026:0:(qsd_reint.c:249:qsd_reint_index()) lustre-OST0000: index version for fid [0x200000005:0x1007:0x0] is 0, but index isn't empty (1)
[ 4585.491268] Lustre: 245026:0:(qsd_reint.c:249:qsd_reint_index()) Skipped 2 previous similar messages
[ 4586.774653] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 4588.414596] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 4589.597140] Lustre: DEBUG MARKER: Write...
[ 4597.339730] LustreError: 246368:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[ 4600.767691] Lustre: DEBUG MARKER: Write...
[ 4604.468549] Lustre: DEBUG MARKER: Write...
[ 4640.976706] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 04:31:51 (1789893111)
[ 4651.028483] LustreError: 250565:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[ 4670.160221] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 04:32:20 (1789893140)
[ 4680.964925] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[ 4681.509501] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[ 4708.780341] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 04:32:58 (1789893178)
[ 4739.770088] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 04:33:28 (1789893208)
[ 4768.473851] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 04:33:57 (1789893237)
[ 4776.364854] Lustre: DEBUG MARKER: User quota (block hardlimit:100 MB)
[ 4791.570858] Lustre: DEBUG MARKER: Write...
[ 4792.373248] Lustre: DEBUG MARKER: Write out of block quota ...
[ 4846.847702] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 04:35:16 (1789893316)
[ 4851.132792] Lustre: DEBUG MARKER: User quota (block hardlimit:1000 MB)
[ 4903.706926] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 04:36:13 (1789893373)
[ 4909.138554] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB)
[ 4920.829533] Lustre: DEBUG MARKER: Write...
[ 4921.670632] Lustre: DEBUG MARKER: Write out of block quota ...
[ 4921.812384] LustreError: 239150:0:(qmt_entry.c:559:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:10240 soft:0 granted:10240 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[ 4953.768704] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 04:37:03 (1789893423)
[ 4967.462418] Lustre: DEBUG MARKER: set to use default quota
[ 4968.343966] Lustre: DEBUG MARKER: set default quota
[ 4969.137267] Lustre: DEBUG MARKER: get default quota
[ 4972.382773] Lustre: DEBUG MARKER: Test not out of quota
[ 4974.144167] Lustre: DEBUG MARKER: Test out of quota
[ 4979.472242] Lustre: DEBUG MARKER: Increase default quota
[ 4996.466986] Lustre: DEBUG MARKER: Set quota to override default quota
[ 4996.498649] LustreError: 237062:0:(qmt_entry.c:559:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45056 time:1790498267 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[ 5001.735497] Lustre: DEBUG MARKER: Set to use default quota again
[ 5013.105950] Lustre: DEBUG MARKER: Cleanup
[ 5046.263372] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 04:38:36 (1789893516)
[ 5058.141733] Lustre: DEBUG MARKER: set default quota for qpool1
[ 5058.920845] Lustre: DEBUG MARKER: Write from user that hasn't lqe
[ 5087.034866] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 04:39:16 (1789893556)
[ 5130.259696] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 04:40:00 (1789893600)
[ 5164.191131] Lustre: DEBUG MARKER: Write...
[ 5165.587067] Lustre: DEBUG MARKER: Write out of block quota ...
[ 5220.742316] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 04:41:30 (1789893690)
[ 5240.681172] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 04:41:50 (1789893710)
[ 5243.650594] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 04:41:53 (1789893713)
[ 5248.288116] Lustre: DEBUG MARKER: sanity-quota test_78A: @@@@@@ FAIL: fallocate did not use quota. kbytes returned:4