[ 2461.555880] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 2463.360438] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 2465.950214] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2466.455706] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 2467.187087] Lustre: Failing over lustre-MDT0000
[ 2467.566615] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2467.569729] Lustre: Skipped 1 previous similar message
[ 2467.570905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 2467.669781] Lustre: server umount lustre-MDT0000 complete
[ 2470.924383] LustreError: 6490:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2470.929391] LustreError: 6490:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages
[ 2481.847200] LDISKFS-fs (dm-0): 7 truncates cleaned up
[ 2481.850063] LDISKFS-fs (dm-0): recovery complete
[ 2481.855235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2481.898571] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2481.985907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2482.025558] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 2483.294769] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2486.277447] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 2487.267439] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo)
[ 2487.270966] Lustre: Skipped 1 previous similar message
[ 2487.276792] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted.
[ 2487.292738] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1778 to 0x280000401:1793)
[ 2487.292809] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1710 to 0x2c0000401:1729)
[ 2488.978150] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2489.492582] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2499.087299] Lustre: DEBUG MARKER: (dd_pid=134452, time=7, timeout=600)
[ 2509.660954] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 2511.608556] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 2514.184432] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2514.679129] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 2515.316667] Lustre: Failing over lustre-MDT0000
[ 2515.470263] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.53@tcp (stopping)
[ 2515.472499] Lustre: Skipped 3 previous similar messages
[ 2515.698573] Lustre: server umount lustre-MDT0000 complete
[ 2515.935688] LustreError: lustre-MDT0000-lwp-MDT0001: operation quota_acquire to node 0@lo failed: rc = -107
[ 2515.938455] LustreError: 6490:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2515.943527] LustreError: 6490:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages
[ 2529.622099] LDISKFS-fs (dm-0): 3 truncates cleaned up
[ 2529.623936] LDISKFS-fs (dm-0): recovery complete
[ 2529.626504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2529.666056] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2529.745667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2531.074668] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2534.908236] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1795 to 0x280000401:1825)
[ 2534.908559] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1710 to 0x2c0000401:1761)
[ 2536.611348] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2537.202132] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2539.651037] Lustre: DEBUG MARKER: (dd_pid=136895, time=0, timeout=600)
[ 2558.948885] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 10:02:48 (1783519368)
[ 2580.765331] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 10:03:09 (1783519389)
[ 2590.006715] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 2593.739296] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 2594.266512] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 2594.793751] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 2595.417028] Lustre: DEBUG MARKER: Set quota for 1 times
[ 2597.088025] Lustre: DEBUG MARKER: Set quota for 2 times
[ 2598.810418] Lustre: DEBUG MARKER: Set quota for 3 times
[ 2600.617501] Lustre: DEBUG MARKER: Set quota for 4 times
[ 2602.283634] Lustre: DEBUG MARKER: Set quota for 5 times
[ 2603.966516] Lustre: DEBUG MARKER: Set quota for 6 times
[ 2605.617990] Lustre: DEBUG MARKER: Set quota for 7 times
[ 2607.362842] Lustre: DEBUG MARKER: Set quota for 8 times
[ 2609.084260] Lustre: DEBUG MARKER: Set quota for 9 times
[ 2610.816378] Lustre: DEBUG MARKER: Set quota for 10 times
[ 2612.442554] Lustre: DEBUG MARKER: Set quota for 11 times
[ 2614.107812] Lustre: DEBUG MARKER: Set quota for 12 times
[ 2615.727487] Lustre: DEBUG MARKER: Set quota for 13 times
[ 2617.375366] Lustre: DEBUG MARKER: Set quota for 14 times
[ 2619.008265] Lustre: DEBUG MARKER: Set quota for 15 times
[ 2620.748441] Lustre: DEBUG MARKER: Set quota for 16 times
[ 2622.470047] Lustre: DEBUG MARKER: Set quota for 17 times
[ 2624.198297] Lustre: DEBUG MARKER: Set quota for 18 times
[ 2642.223846] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 10:04:11 (1783519451)
[ 2652.639792] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 2652.640745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 2652.643781] LustreError: Skipped 1 previous similar message
[ 2654.038945] Lustre: server umount lustre-MDT0000 complete
[ 2655.271929] LustreError: 6474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783519465 with bad export cookie 9710815106719801994
[ 2655.274133] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2655.275575] LustreError: 6474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 2655.403948] Lustre: server umount lustre-MDT0001 complete
[ 2666.945086] Lustre: server umount lustre-OST0000 complete
[ 2678.443230] Lustre: server umount lustre-OST0001 complete
[ 2683.708971] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local
[ 2687.743915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2687.905956] LustreError: 164347:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2687.914597] LustreError: 164347:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages
[ 2687.939033] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2689.421106] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2692.430822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2693.986102] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2694.932267] Lustre: 165502:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 2697.059317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 2698.209793] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1827 to 0x280000401:1857)
[ 2699.037516] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2701.829226] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 2702.946506] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97)
[ 2702.947307] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1764 to 0x2c0000401:1793)
[ 2703.902341] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2707.139488] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 2708.379622] Lustre: 167378:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 2717.330487] Lustre: server umount lustre-MDT0000 complete
[ 2718.175457] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 2718.535665] LustreError: 164328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783519528 with bad export cookie 9710815106719810478
[ 2718.537294] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2718.539574] LustreError: 164328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages
[ 2718.687804] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping)
[ 2718.689807] Lustre: Skipped 4 previous similar messages
[ 2720.669942] Lustre: server umount lustre-MDT0001 complete
[ 2721.924593] Lustre: server umount lustre-OST0000 complete
[ 2723.148398] Lustre: server umount lustre-OST0001 complete
[ 2728.179809] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local
[ 2731.496850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2731.663412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2731.665907] Lustre: Skipped 3 previous similar messages
[ 2732.929035] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2735.636275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2737.025898] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2737.945187] Lustre: 171159:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 2740.035415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 2741.219284] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1827 to 0x280000401:1889)
[ 2741.975830] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2744.681316] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 2745.763369] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1764 to 0x2c0000401:1825)
[ 2745.775977] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:129)
[ 2746.649613] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2749.672722] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 2750.790651] Lustre: 173025:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 2753.710104] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 10:06:02 (1783519562)
[ 2754.188910] Lustre: DEBUG MARKER: OST0_SIZE: 3605396 required: 6144
[ 2754.621510] Lustre: DEBUG MARKER: run for 4MB test file
[ 2759.451210] Lustre: DEBUG MARKER: User quota (limit: 4 MB)
[ 2761.420765] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT
[ 2761.965830] Lustre: DEBUG MARKER: Write half of file
[ 2762.566873] Lustre: DEBUG MARKER: Write out of block quota ...
[ 2763.132528] Lustre: DEBUG MARKER: Step1: done
[ 2763.669190] Lustre: DEBUG MARKER: Step2: rewrite should succeed
[ 2764.234351] Lustre: DEBUG MARKER: Step2: done
[ 2777.407682] Lustre: DEBUG MARKER: OST0_SIZE: 3605396 required: 61440
[ 2777.897953] Lustre: DEBUG MARKER: run for 40MB test file
[ 2781.544678] Lustre: DEBUG MARKER: User quota (limit: 40 MB)
[ 2783.509995] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT
[ 2784.057402] Lustre: DEBUG MARKER: Write half of file
[ 2784.908977] Lustre: DEBUG MARKER: Write out of block quota ...
[ 2785.741365] Lustre: DEBUG MARKER: Step1: done
[ 2786.241971] Lustre: DEBUG MARKER: Step2: rewrite should succeed
[ 2786.774495] Lustre: DEBUG MARKER: Step2: done
[ 2811.138359] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 10:07:00 (1783519620)
[ 2832.299827] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 10:07:21 (1783519641)
[ 2844.954732] Lustre: DEBUG MARKER: Write...
[ 2845.732996] Lustre: DEBUG MARKER: Write out of block quota ...
[ 2877.712648] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 10:08:06 (1783519686)
[ 2879.729881] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 10:08:08 (1783519688)
[ 2883.988990] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 10:08:13 (1783519693)
[ 2887.379438] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 10:08:16 (1783519696)
[ 2890.598882] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 10:08:19 (1783519699)
[ 2919.947701] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 2972.686875] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 3041.772487] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 10:10:50 (1783519850)
[ 3064.352350] Lustre: DEBUG MARKER: Restart...
[ 3068.385379] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 3068.386529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 3068.389236] Lustre: Skipped 19 previous similar messages
[ 3068.392288] Lustre: Skipped 3 previous similar messages
[ 3071.701675] Lustre: server umount lustre-MDT0000 complete
[ 3072.973654] LustreError: 173028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783519882 with bad export cookie 9710815106719812767
[ 3072.977374] LustreError: 173028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 3072.977676] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3073.097241] Lustre: server umount lustre-MDT0001 complete
[ 3084.609041] Lustre: server umount lustre-OST0000 complete
[ 3095.061450] Lustre: server umount lustre-OST0001 complete
[ 3099.968759] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local
[ 3103.522538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3103.678939] LustreError: 197785:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3103.686128] LustreError: 197785:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages
[ 3103.705166] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3103.708093] Lustre: Skipped 3 previous similar messages
[ 3105.112337] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3107.907536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3109.458704] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3110.430431] Lustre: 198927:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 3112.728375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 3113.890045] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1900 to 0x280000401:1921)
[ 3114.797693] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3117.580681] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 3118.690198] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:161)
[ 3118.690793] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1834 to 0x2c0000401:1857)
[ 3119.566981] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3122.608089] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 3123.768746] Lustre: 200798:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 3149.500363] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 10:12:38 (1783519958)
[ 3174.050362] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 10:13:03 (1783519983)
[ 3232.259271] hrtimer: interrupt took 2391152 ns
[ 4030.754542] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 10:27:19 (1783520839)
[ 4035.040409] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 4035.043390] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 4035.048676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 4040.217573] Lustre: server umount lustre-MDT0000 complete
[ 4041.184141] LustreError: 198535:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 4041.190631] LustreError: 198535:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages
[ 4041.724476] LustreError: 197767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783520851 with bad export cookie 9710815106719821664
[ 4041.727890] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4041.729876] LustreError: 197767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 4041.856735] Lustre: server umount lustre-MDT0001 complete
[ 4053.538304] Lustre: server umount lustre-OST0000 complete
[ 4065.399362] Lustre: server umount lustre-OST0001 complete
[ 4071.931089] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local
[ 4076.148489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4076.320859] LustreError: 208614:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 4076.354743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4076.356806] Lustre: Skipped 3 previous similar messages
[ 4077.955287] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4081.299838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4083.117539] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4084.422873] Lustre: 209756:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 4087.231202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 4087.404829] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 4087.407460] Lustre: Skipped 1 previous similar message
[ 4089.754149] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4091.491594] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:193)
[ 4093.024197] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 4095.350746] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4101.611259] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:6859 to 0x2c0000401:6881)
[ 4101.611326] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:6923 to 0x280000401:6945)
[ 4104.418710] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 4105.866858] Lustre: 211627:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 4122.205332] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 10:28:51 (1783520931)
[ 4137.875268] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 10:29:06 (1783520946)
[ 4152.782358] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 10:29:21 (1783520961)
[ 4170.160753] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 10:29:39 (1783520979)
[ 4188.520982] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 10:29:57 (1783520997)
[ 4229.372324] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 10:30:38 (1783521038)
[ 4243.413696] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 10:30:52 (1783521052)
[ 4280.861799] LustreError: 210116:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -3, flags:0x1  qsd:lustre-OST0001 qtype:grp lqe: ffff8a210d559800 id:60000 enforced:1 granted: 0 pending:0 waiting:1032 req:1 usage: 0 qunit:0 qtune:0 edquot:0 default:no revoke:0
[ 4280.876297] LustreError: 210116:0:(qsd_handler.c:773:qsd_op_begin0()) $$$ ID isn't enforced on master, it probably due to a legeal race, if this message is showing up constantly, there could be some inconsistence between master & slave, and quota reintegration needs be re-triggered.  qsd:lustre-OST0001 qtype:grp lqe: ffff8a210d559800 id:60000 enforced:1 granted: 0 pending:0 waiting:0 req:0 usage: 0 qunit:0 qtune:0 edquot:0 default:no revoke:0
[ 4342.753306] LustreError: 3623:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -3, flags:0x8  qsd:lustre-OST0001 qtype:grp lqe: ffff8a210d559800 id:60000 enforced:1 granted: 0 pending:0 waiting:0 req:1 usage: 1024 qunit:0 qtune:0 edquot:0 default:no revoke:0
[ 4342.761609] LustreError: 3623:0:(qsd_handler.c:298:qsd_req_completion()) Skipped 1 previous similar message
[ 4343.855174] Lustre: DEBUG MARKER: sanity-quota test_48: @@@@@@ FAIL: QMT: mds1 qid 60000 is not deleted