[ 2531.002964] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 2533.009467] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 2535.621339] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2536.153235] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 2536.839072] Lustre: Failing over lustre-MDT0000
[ 2537.132888] LustreError: lustre-MDT0000-lwp-OST0000: operation quota_acquire to node 0@lo failed: rc = -107
[ 2537.136969] LustreError: Skipped 1 previous similar message
[ 2537.139423] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2537.144691] Lustre: Skipped 1 previous similar message
[ 2537.146851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 2537.218925] Lustre: server umount lustre-MDT0000 complete
[ 2540.511771] LustreError: 6483:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2540.516829] LustreError: 6483:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages
[ 2551.194657] LDISKFS-fs (dm-0): 7 truncates cleaned up
[ 2551.196036] LDISKFS-fs (dm-0): recovery complete
[ 2551.200995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2551.239287] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2551.328153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2551.365810] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 2552.126698] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 2552.694410] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2556.385753] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 2556.388146] Lustre: Skipped 1 previous similar message
[ 2556.400361] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted.
[ 2556.419660] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1828 to 0x2c0000401:1857)
[ 2556.419892] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1832 to 0x280000401:1857)
[ 2557.517812] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2558.017091] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2569.629037] Lustre: DEBUG MARKER: (dd_pid=127353, time=9, timeout=600)
[ 2578.715415] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 2580.584490] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 2583.217333] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2583.704519] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 2584.349211] Lustre: Failing over lustre-MDT0000
[ 2584.708106] Lustre: server umount lustre-MDT0000 complete
[ 2585.567493] LustreError: lustre-MDT0000-lwp-MDT0001: operation quota_acquire to node 0@lo failed: rc = -107
[ 2598.745513] LDISKFS-fs (dm-0): 2 truncates cleaned up
[ 2598.746897] LDISKFS-fs (dm-0): recovery complete
[ 2598.749616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2598.787881] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2598.875837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2600.279203] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2604.026545] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1859 to 0x280000401:1889)
[ 2604.026775] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1828 to 0x2c0000401:1889)
[ 2605.205099] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2605.760279] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2608.204599] Lustre: DEBUG MARKER: (dd_pid=129726, time=0, timeout=600)
[ 2622.075493] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 22:21:00 (1772680860)
[ 2644.112201] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 22:21:22 (1772680882)
[ 2653.374280] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 2657.025552] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 2657.529381] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 2658.025952] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 2658.619790] Lustre: DEBUG MARKER: Set quota for 1 times
[ 2660.256237] Lustre: DEBUG MARKER: Set quota for 2 times
[ 2661.877210] Lustre: DEBUG MARKER: Set quota for 3 times
[ 2663.463188] Lustre: DEBUG MARKER: Set quota for 4 times
[ 2665.152354] Lustre: DEBUG MARKER: Set quota for 5 times
[ 2666.776124] Lustre: DEBUG MARKER: Set quota for 6 times
[ 2668.488706] Lustre: DEBUG MARKER: Set quota for 7 times
[ 2670.044232] Lustre: DEBUG MARKER: Set quota for 8 times
[ 2671.785777] Lustre: DEBUG MARKER: Set quota for 9 times
[ 2673.435228] Lustre: DEBUG MARKER: Set quota for 10 times
[ 2675.034617] Lustre: DEBUG MARKER: Set quota for 11 times
[ 2676.713213] Lustre: DEBUG MARKER: Set quota for 12 times
[ 2678.328761] Lustre: DEBUG MARKER: Set quota for 13 times
[ 2680.029820] Lustre: DEBUG MARKER: Set quota for 14 times
[ 2681.618314] Lustre: DEBUG MARKER: Set quota for 15 times
[ 2683.350480] Lustre: DEBUG MARKER: Set quota for 16 times
[ 2684.986210] Lustre: DEBUG MARKER: Set quota for 17 times
[ 2686.667456] Lustre: DEBUG MARKER: Set quota for 18 times
[ 2688.392482] Lustre: DEBUG MARKER: Set quota for 19 times
[ 2706.184716] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 22:22:24 (1772680944)
[ 2711.683277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 2711.687108] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 2711.689621] LustreError: Skipped 1 previous similar message
[ 2717.152057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 2717.154871] Lustre: Skipped 6 previous similar messages
[ 2717.910227] Lustre: server umount lustre-MDT0000 complete
[ 2719.101719] LustreError: 9436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772680958 with bad export cookie 16221676005816137451
[ 2719.103798] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2719.105569] LustreError: 9436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 2719.215943] Lustre: server umount lustre-MDT0001 complete
[ 2730.500821] Lustre: server umount lustre-OST0000 complete
[ 2740.975616] Lustre: server umount lustre-OST0001 complete
[ 2745.913924] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local
[ 2749.617072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2749.764310] LustreError: 157519:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2749.769450] LustreError: 157519:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 26 previous similar messages
[ 2749.793034] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2751.062693] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2753.832620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2755.204736] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2756.130861] Lustre: 158638:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 2758.250305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 2759.393930] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1891 to 0x280000401:1921)
[ 2760.134985] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2762.976860] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 2764.066186] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1892 to 0x2c0000401:1921)
[ 2764.066745] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97)
[ 2764.909782] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2767.962330] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 2769.078736] Lustre: 160487:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 2784.735651] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 2784.736312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 2784.741205] Lustre: Skipped 3 previous similar messages
[ 2788.382878] Lustre: server umount lustre-MDT0000 complete
[ 2789.583512] LustreError: 157500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772681028 with bad export cookie 16221676005816146992
[ 2789.585407] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2789.586306] LustreError: 157500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 2789.687537] Lustre: server umount lustre-MDT0001 complete
[ 2801.153618] Lustre: server umount lustre-OST0000 complete
[ 2812.626285] Lustre: server umount lustre-OST0001 complete
[ 2817.911608] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local
[ 2821.419542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2821.606700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2821.608388] Lustre: Skipped 3 previous similar messages
[ 2822.865963] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2825.650616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2827.056591] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2828.112392] Lustre: 164267:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 2830.248964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 2831.393950] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1891 to 0x280000401:1953)
[ 2832.149062] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2835.083078] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 2836.195313] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:129)
[ 2836.195758] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1892 to 0x2c0000401:1953)
[ 2836.995598] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2840.318370] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 2841.495021] Lustre: 166111:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 2849.768590] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 22:24:48 (1772681088)
[ 2850.304335] Lustre: DEBUG MARKER: OST0_SIZE: 3605396 required: 6144
[ 2850.905993] Lustre: DEBUG MARKER: run for 4MB test file
[ 2856.116715] Lustre: DEBUG MARKER: User quota (limit: 4 MB)
[ 2857.913203] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT
[ 2858.406325] Lustre: DEBUG MARKER: Write half of file
[ 2859.001713] Lustre: DEBUG MARKER: Write out of block quota ...
[ 2859.545657] Lustre: DEBUG MARKER: Step1: done
[ 2860.072970] Lustre: DEBUG MARKER: Step2: rewrite should succeed
[ 2860.567900] Lustre: DEBUG MARKER: Step2: done
[ 2878.618296] Lustre: DEBUG MARKER: OST0_SIZE: 3605396 required: 61440
[ 2879.149396] Lustre: DEBUG MARKER: run for 40MB test file
[ 2882.811578] Lustre: DEBUG MARKER: User quota (limit: 40 MB)
[ 2884.682291] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT
[ 2885.158050] Lustre: DEBUG MARKER: Write half of file
[ 2886.001401] Lustre: DEBUG MARKER: Write out of block quota ...
[ 2886.808829] Lustre: DEBUG MARKER: Step1: done
[ 2887.301394] Lustre: DEBUG MARKER: Step2: rewrite should succeed
[ 2887.820977] Lustre: DEBUG MARKER: Step2: done
[ 2912.609602] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 22:25:51 (1772681151)
[ 2933.677732] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 22:26:12 (1772681172)
[ 2946.435485] Lustre: DEBUG MARKER: Write...
[ 2947.113327] Lustre: DEBUG MARKER: Write out of block quota ...
[ 2980.027056] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 22:26:58 (1772681218)
[ 2982.010539] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 22:27:00 (1772681220)
[ 2986.299762] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 22:27:04 (1772681224)
[ 2989.644637] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 22:27:08 (1772681228)
[ 2993.339885] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 22:27:11 (1772681231)
[ 3020.204355] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 3075.903188] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 3163.241550] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 22:30:01 (1772681401)
[ 3188.057221] Lustre: DEBUG MARKER: Restart...
[ 3189.215595] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 3189.218991] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 3189.224380] Lustre: Skipped 15 previous similar messages
[ 3189.226632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 3195.225867] Lustre: server umount lustre-MDT0000 complete
[ 3196.623506] LustreError: 174405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772681435 with bad export cookie 16221676005816149295
[ 3196.626975] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3196.629345] LustreError: 174405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 3196.759678] Lustre: server umount lustre-MDT0001 complete
[ 3204.063248] LustreError: 3633:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED  req@ffff9d2db4ab1500 x1858787931776000/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 3204.068941] LustreError: 3633:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8  qsd:lustre-OST0000 qtype:usr lqe: ffff9d2db7d88480 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 1596 qunit:0 qtune:0 edquot:0 default:no revoke:0
[ 3208.208752] Lustre: server umount lustre-OST0000 complete
[ 3218.836274] Lustre: server umount lustre-OST0001 complete
[ 3224.686526] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local
[ 3228.193246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3228.376273] LustreError: 191131:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3228.380591] LustreError: 191131:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages
[ 3228.409222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3228.412933] Lustre: Skipped 3 previous similar messages
[ 3229.700525] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3232.551832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3234.036766] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3235.000415] Lustre: 192239:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 3237.338739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 3238.498991] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1965 to 0x280000401:1985)
[ 3239.226027] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3242.124319] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 3243.298266] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:161)
[ 3243.298348] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1961 to 0x2c0000401:1985)
[ 3244.163429] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3247.452306] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 3248.568481] Lustre: 194084:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 3279.689425] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 22:31:58 (1772681518)
[ 3309.596367] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 22:32:27 (1772681547)
[ 3837.512762] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 22:41:15 (1772682075)
[ 3842.016137] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 3842.020803] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 3842.028042] Lustre: Skipped 3 previous similar messages
[ 3842.031311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 3842.035347] Lustre: Skipped 7 previous similar messages
[ 3843.039375] LustreError: 3635:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED  req@ffff9d2c8373ce00 x1858787934677504/t0(0) o601->lustre-MDT0000-lwp-MDT0000@0@lo:23/10 lens 336/336 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 3843.053149] LustreError: 3635:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8  qsd:lustre-MDT0000 qtype:usr lqe: ffff9d2c84a0ca80 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 2532 qunit:0 qtune:0 edquot:0 default:no revoke:0
[ 3847.550859] Lustre: server umount lustre-MDT0000 complete
[ 3847.649275] LustreError: 192790:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3847.654950] LustreError: 192790:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages
[ 3849.193653] LustreError: 191111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772682088 with bad export cookie 16221676005816158073
[ 3849.196730] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3849.199073] LustreError: 191111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 3849.353597] Lustre: server umount lustre-MDT0001 complete
[ 3851.231350] LustreError: 3633:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED  req@ffff9d2c8e8f7480 x1858787934680192/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 3851.240129] LustreError: 3633:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8  qsd:lustre-OST0000 qtype:usr lqe: ffff9d2db7d88480 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 1596 qunit:0 qtune:0 edquot:0 default:no revoke:0
[ 3861.037877] Lustre: server umount lustre-OST0000 complete
[ 3872.783119] Lustre: server umount lustre-OST0001 complete
[ 3878.642144] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local
[ 3882.834837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3883.080222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3883.083224] Lustre: Skipped 3 previous similar messages
[ 3884.651102] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3888.172572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3889.795583] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3890.730740] Lustre: 202913:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 3893.558687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 3895.797420] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3897.827850] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:193)
[ 3899.850317] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 3902.568104] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3908.259659] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:6988 to 0x280000401:7009)
[ 3908.259973] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:6986 to 0x2c0000401:7009)
[ 3912.026930] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 3913.789336] Lustre: 204764:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 3934.092610] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 22:42:52 (1772682172)
[ 3950.094987] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 22:43:08 (1772682188)
[ 3964.565713] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 22:43:23 (1772682203)
[ 3980.153089] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 22:43:38 (1772682218)
[ 3995.873442] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 22:43:54 (1772682234)
[ 4036.005710] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 22:44:34 (1772682274)
[ 4050.533764] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 22:44:48 (1772682288)
[ 4077.913093] LustreError: 203274:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -3, flags:0x1  qsd:lustre-OST0000 qtype:grp lqe: ffff9d2c881d4f00 id:60000 enforced:1 granted: 0 pending:0 waiting:1032 req:1 usage: 0 qunit:0 qtune:0 edquot:0 default:no revoke:0
[ 4077.921064] LustreError: 203274:0:(qsd_handler.c:772:qsd_op_begin0()) $$$ ID isn't enforced on master, it probably due to a legeal race, if this message is showing up constantly, there could be some inconsistence between master & slave, and quota reintegration needs be re-triggered.  qsd:lustre-OST0000 qtype:grp lqe: ffff9d2c881d4f00 id:60000 enforced:1 granted: 0 pending:0 waiting:0 req:0 usage: 0 qunit:0 qtune:0 edquot:0 default:no revoke:0
[ 4137.953085] LustreError: 3634:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -3, flags:0x8  qsd:lustre-OST0000 qtype:grp lqe: ffff9d2c881d4f00 id:60000 enforced:1 granted: 0 pending:0 waiting:0 req:1 usage: 1024 qunit:0 qtune:0 edquot:0 default:no revoke:0
[ 4137.967343] LustreError: 3634:0:(qsd_handler.c:298:qsd_req_completion()) Skipped 1 previous similar message
[ 4150.214774] Lustre: DEBUG MARKER: sanity-quota test_48: @@@@@@ FAIL: QMT: mds1 qid 60000 is not deleted