[ 2536.528608] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 2538.391285] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 2540.887563] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2541.393501] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 2542.062895] Lustre: Failing over lustre-MDT0000
[ 2542.462356] LustreError: lustre-MDT0000-lwp-OST0000: operation quota_acquire to node 0@lo failed: rc = -107
[ 2542.465193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 2542.546543] Lustre: server umount lustre-MDT0000 complete
[ 2544.608870] LustreError: 8037:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2544.614448] LustreError: 8037:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages
[ 2556.563700] LDISKFS-fs (dm-0): 7 truncates cleaned up
[ 2556.564901] LDISKFS-fs (dm-0): recovery complete
[ 2556.568239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2556.602852] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2556.688841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2556.732595] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 2557.154480] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 2557.981630] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2562.030764] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted.
[ 2562.046722] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1988 to 0x280000401:2017)
[ 2562.046885] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1919 to 0x2c0000401:1953)
[ 2563.159939] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2563.680692] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2573.237224] Lustre: DEBUG MARKER: (dd_pid=133664, time=7, timeout=600)
[ 2585.288586] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 2587.069900] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 2589.674265] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2590.187766] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 2590.805406] Lustre: Failing over lustre-MDT0000
[ 2591.190781] Lustre: server umount lustre-MDT0000 complete
[ 2592.737258] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 2592.739355] LustreError: Skipped 1 previous similar message
[ 2605.101603] LDISKFS-fs (dm-0): 3 truncates cleaned up
[ 2605.102754] LDISKFS-fs (dm-0): recovery complete
[ 2605.105573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2605.143561] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2605.229251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2606.456552] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2610.689749] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2019 to 0x280000401:2049)
[ 2610.689773] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1919 to 0x2c0000401:1985)
[ 2611.825466] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2612.384513] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2614.837972] Lustre: DEBUG MARKER: (dd_pid=136127, time=0, timeout=600)
[ 2629.758696] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 20:38:53 (1777595933)
[ 2650.836136] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 20:39:14 (1777595954)
[ 2659.771712] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 2663.232949] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 2663.707670] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 2664.185671] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 2664.732445] Lustre: DEBUG MARKER: Set quota for 1 times
[ 2666.405263] Lustre: DEBUG MARKER: Set quota for 2 times
[ 2667.980336] Lustre: DEBUG MARKER: Set quota for 3 times
[ 2669.598714] Lustre: DEBUG MARKER: Set quota for 4 times
[ 2671.232601] Lustre: DEBUG MARKER: Set quota for 5 times
[ 2672.792391] Lustre: DEBUG MARKER: Set quota for 6 times
[ 2674.402891] Lustre: DEBUG MARKER: Set quota for 7 times
[ 2676.043610] Lustre: DEBUG MARKER: Set quota for 8 times
[ 2677.703687] Lustre: DEBUG MARKER: Set quota for 9 times
[ 2679.401189] Lustre: DEBUG MARKER: Set quota for 10 times
[ 2681.091847] Lustre: DEBUG MARKER: Set quota for 11 times
[ 2682.826659] Lustre: DEBUG MARKER: Set quota for 12 times
[ 2684.545800] Lustre: DEBUG MARKER: Set quota for 13 times
[ 2686.218419] Lustre: DEBUG MARKER: Set quota for 14 times
[ 2687.858913] Lustre: DEBUG MARKER: Set quota for 15 times
[ 2689.526584] Lustre: DEBUG MARKER: Set quota for 16 times
[ 2691.145642] Lustre: DEBUG MARKER: Set quota for 17 times
[ 2692.724264] Lustre: DEBUG MARKER: Set quota for 18 times
[ 2694.373728] Lustre: DEBUG MARKER: Set quota for 19 times
[ 2723.510692] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 20:40:26 (1777596026)
[ 2731.929427] Lustre: server umount lustre-MDT0000 complete
[ 2733.124918] LustreError: 6473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777596036 with bad export cookie 6858775469265776877
[ 2733.127251] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2733.129749] LustreError: 6473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 2733.240913] Lustre: server umount lustre-MDT0001 complete
[ 2749.280211] Lustre: 3635:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777596036/real 1777596036]  req@ffff88e089e4df80 x1863941737126272/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1777596052 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2749.280278] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2749.299549] Lustre: 3635:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 2749.306304] Lustre: Skipped 9 previous similar messages
[ 2754.464266] Lustre: 3635:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777596041/real 1777596041]  req@ffff88df88dd4700 x1863941737126912/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1777596057 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2754.482990] Lustre: 3635:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 2755.551180] Lustre: server umount lustre-OST0000 complete
[ 2757.540232] Lustre: server umount lustre-OST0001 complete
[ 2763.171736] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local
[ 2767.291809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2767.432591] LustreError: 162247:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2767.437672] LustreError: 162247:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 25 previous similar messages
[ 2767.461846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2768.725657] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2772.019448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2772.167815] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 2773.434319] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2774.842704] Lustre: 163367:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 2777.759347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 2777.897397] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 2780.028210] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2781.990944] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97)
[ 2783.791939] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 2786.152253] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2792.418977] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2052 to 0x280000401:2081)
[ 2792.419308] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1987 to 0x2c0000401:2017)
[ 2795.097201] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 2796.411577] Lustre: 165206:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 2807.779218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 2807.785353] Lustre: Skipped 3 previous similar messages
[ 2812.897467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 2812.899514] Lustre: Skipped 3 previous similar messages
[ 2813.100541] Lustre: server umount lustre-MDT0000 complete
[ 2814.702252] LustreError: 162230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777596117 with bad export cookie 6858775469265786908
[ 2814.705668] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2814.711387] LustreError: 162230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 2814.877927] Lustre: server umount lustre-MDT0001 complete
[ 2826.265009] Lustre: server umount lustre-OST0000 complete
[ 2838.121842] Lustre: server umount lustre-OST0001 complete
[ 2843.972016] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local
[ 2848.083188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2848.295184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2848.297618] Lustre: Skipped 1 previous similar message
[ 2849.712908] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2853.615375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2855.476208] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2856.842705] Lustre: 168989:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 2859.236186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 2861.194970] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2864.164712] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 2865.251910] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:129)
[ 2866.020209] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2873.445343] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2052 to 0x280000401:2113)
[ 2873.446050] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1987 to 0x2c0000401:2049)
[ 2880.440611] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 2882.130970] Lustre: 170847:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 2886.262972] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 20:43:09 (1777596189)
[ 2886.955885] Lustre: DEBUG MARKER: OST0_SIZE: 3605396 required: 6144
[ 2887.638494] Lustre: DEBUG MARKER: run for 4MB test file
[ 2893.016877] Lustre: DEBUG MARKER: User quota (limit: 4 MB)
[ 2895.114928] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT
[ 2895.742656] Lustre: DEBUG MARKER: Write half of file
[ 2896.365383] Lustre: DEBUG MARKER: Write out of block quota ...
[ 2897.036268] Lustre: DEBUG MARKER: Step1: done
[ 2897.560862] Lustre: DEBUG MARKER: Step2: rewrite should succeed
[ 2898.212035] Lustre: DEBUG MARKER: Step2: done
[ 2916.418492] Lustre: DEBUG MARKER: OST0_SIZE: 3605396 required: 61440
[ 2917.092647] Lustre: DEBUG MARKER: run for 40MB test file
[ 2921.680743] Lustre: DEBUG MARKER: User quota (limit: 40 MB)
[ 2924.179892] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT
[ 2924.936851] Lustre: DEBUG MARKER: Write half of file
[ 2926.330400] Lustre: DEBUG MARKER: Write out of block quota ...
[ 2927.468133] Lustre: DEBUG MARKER: Step1: done
[ 2928.013844] Lustre: DEBUG MARKER: Step2: rewrite should succeed
[ 2928.590775] Lustre: DEBUG MARKER: Step2: done
[ 2951.918028] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 20:44:15 (1777596255)
[ 2970.746616] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 20:44:34 (1777596274)
[ 2985.212838] Lustre: DEBUG MARKER: Write...
[ 2986.017550] Lustre: DEBUG MARKER: Write out of block quota ...
[ 3016.837788] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 20:45:20 (1777596320)
[ 3019.062747] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 20:45:22 (1777596322)
[ 3023.573132] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 20:45:27 (1777596327)
[ 3026.867426] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 20:45:30 (1777596330)
[ 3029.930625] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 20:45:33 (1777596333)
[ 3057.151266] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 3113.965927] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 3194.926814] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 20:48:18 (1777596498)
[ 3216.276773] Lustre: DEBUG MARKER: Restart...
[ 3217.455612] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 3217.459237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 3221.985307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 3221.987238] Lustre: Skipped 2 previous similar messages
[ 3223.582244] Lustre: server umount lustre-MDT0000 complete
[ 3224.716282] LustreError: 167864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777596527 with bad export cookie 6858775469265789211
[ 3224.719978] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3224.720114] LustreError: 167864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 3224.839766] Lustre: server umount lustre-MDT0001 complete
[ 3236.351749] Lustre: server umount lustre-OST0000 complete
[ 3240.928243] LustreError: 3634:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED  req@ffff88df8d329180 x1863941737496192/t0(0) o601->lustre-MDT0000-lwp-OST0001@0@lo:23/10 lens 336/336 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 3240.938923] LustreError: 3634:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8  qsd:lustre-OST0001 qtype:usr lqe: ffff88df86187200 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 1596 qunit:0 qtune:0 edquot:0 default:no revoke:0
[ 3247.827215] Lustre: server umount lustre-OST0001 complete
[ 3252.561267] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local
[ 3256.310958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3256.477761] LustreError: 195925:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3256.483406] LustreError: 195925:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages
[ 3256.508625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3256.510740] Lustre: Skipped 3 previous similar messages
[ 3257.868548] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3260.909196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3262.438904] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3263.471573] Lustre: 197034:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 3265.913279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 3267.046433] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2125 to 0x280000401:2145)
[ 3268.069710] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3271.139582] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:161)
[ 3271.178348] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 3273.281720] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3276.260090] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2057 to 0x2c0000401:2081)
[ 3277.209131] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 3278.844811] Lustre: 198874:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 3307.008865] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 20:50:10 (1777596610)
[ 3337.267485] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 20:50:40 (1777596640)
[ 3802.942194] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 20:58:26 (1777597106)
[ 3808.737114] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 3808.738561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 3808.744046] Lustre: Skipped 11 previous similar messages
[ 3808.748338] Lustre: Skipped 3 previous similar messages
[ 3810.272264] LustreError: 3634:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED  req@ffff88df8a093480 x1863941740374912/t0(0) o601->lustre-MDT0000-lwp-MDT0000@0@lo:23/10 lens 336/336 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 3810.281097] LustreError: 3634:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8  qsd:lustre-MDT0000 qtype:usr lqe: ffff88df92657e00 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 2548 qunit:0 qtune:0 edquot:0 default:no revoke:0
[ 3812.085158] Lustre: server umount lustre-MDT0000 complete
[ 3813.197795] LustreError: 195906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777597116 with bad export cookie 6858775469265798101
[ 3813.200044] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3813.201033] LustreError: 195906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 3813.304133] Lustre: server umount lustre-MDT0001 complete
[ 3818.464303] LustreError: 3634:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED  req@ffff88e0aff83100 x1863941740376704/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 3818.471476] LustreError: 3634:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8  qsd:lustre-OST0000 qtype:usr lqe: ffff88e0bd706600 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 1596 qunit:0 qtune:0 edquot:0 default:no revoke:0
[ 3824.719637] Lustre: server umount lustre-OST0000 complete
[ 3826.656293] LustreError: 3637:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED  req@ffff88df8a090a80 x1863941740377344/t0(0) o601->lustre-MDT0000-lwp-OST0001@0@lo:23/10 lens 336/336 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 3826.667646] LustreError: 3637:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8  qsd:lustre-OST0001 qtype:usr lqe: ffff88df87167380 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 1596 qunit:0 qtune:0 edquot:0 default:no revoke:0
[ 3836.127085] Lustre: server umount lustre-OST0001 complete
[ 3840.995728] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local
[ 3844.354434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3844.496880] LustreError: 206614:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3844.502683] LustreError: 206614:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages
[ 3844.525044] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3844.527018] Lustre: Skipped 3 previous similar messages
[ 3845.709177] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3848.260915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3849.607865] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3850.503537] Lustre: 207723:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 3852.494812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 3854.302339] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3854.691483] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7148 to 0x280000401:7169)
[ 3857.135953] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 3858.210909] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:193)
[ 3858.211136] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7082 to 0x2c0000401:7105)
[ 3858.905047] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3862.081308] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 3877.906713] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 20:59:41 (1777597181)
[ 3890.658854] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 20:59:54 (1777597194)
[ 3903.695969] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 21:00:07 (1777597207)
[ 3919.730836] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 21:00:23 (1777597223)
[ 3934.579613] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 21:00:38 (1777597238)
[ 3976.614468] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 21:01:20 (1777597280)
[ 3990.623866] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 21:01:34 (1777597294)
[ 4028.152863] LustreError: 208081:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -3, flags:0x1  qsd:lustre-OST0000 qtype:grp lqe: ffff88df93489200 id:60000 enforced:1 granted: 0 pending:0 waiting:1032 req:1 usage: 0 qunit:0 qtune:0 edquot:0 default:no revoke:0
[ 4028.165217] LustreError: 208081:0:(qsd_handler.c:772:qsd_op_begin0()) $$$ ID isn't enforced on master, it probably due to a legeal race, if this message is showing up constantly, there could be some inconsistence between master & slave, and quota reintegration needs be re-triggered.  qsd:lustre-OST0000 qtype:grp lqe: ffff88df93489200 id:60000 enforced:1 granted: 0 pending:0 waiting:0 req:0 usage: 0 qunit:0 qtune:0 edquot:0 default:no revoke:0
[ 4088.803316] LustreError: 3637:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -3, flags:0x8  qsd:lustre-OST0000 qtype:grp lqe: ffff88df93489200 id:60000 enforced:1 granted: 0 pending:0 waiting:0 req:1 usage: 1024 qunit:0 qtune:0 edquot:0 default:no revoke:0
[ 4088.812846] LustreError: 3637:0:(qsd_handler.c:298:qsd_req_completion()) Skipped 1 previous similar message
[ 4090.594637] Lustre: DEBUG MARKER: sanity-quota test_48: @@@@@@ FAIL: QMT: mds1 qid 60000 is not deleted