[ 5883.465977] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5891.586421] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 5903.060240] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5905.803765] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5909.345893] Lustre: Failing over lustre-MDT0000
[ 5909.469939] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.31@tcp (stopping)
[ 5909.502492] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5909.514302] Lustre: Skipped 2 previous similar messages
[ 5910.001382] Lustre: server umount lustre-MDT0000 complete
[ 5914.509452] LustreError: 23607:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5914.542973] LustreError: 23607:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages
[ 5930.975208] Lustre: 3650:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789055823/real 1789055823]  req@ffff8a684e38bb80 x1875954918414592/t0(0) o400->MGC192.168.204.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789055839 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5930.994107] Lustre: 3650:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
[ 5930.997502] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5934.083931] LDISKFS-fs (dm-0): 7 truncates cleaned up
[ 5934.087037] LDISKFS-fs (dm-0): recovery complete
[ 5934.104093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5941.606465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5941.687970] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5942.160114] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 5946.221302] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5946.854170] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 5946.911142] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted.
[ 5946.980780] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:504 to 0x2c0000401:545)
[ 5946.983514] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:571 to 0x280000401:609)
[ 5956.105631] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5958.068150] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5964.122640] Lustre: DEBUG MARKER: (dd_pid=129485, time=0, timeout=600)
[ 5988.412730] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5995.381821] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 6003.224223] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 6004.752253] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 6006.787971] Lustre: Failing over lustre-MDT0000
[ 6007.232986] Lustre: server umount lustre-MDT0000 complete
[ 6008.287804] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 6008.291207] LustreError: 7806:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6008.294060] LustreError: Skipped 2 previous similar messages
[ 6008.319307] LustreError: 7806:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 36 previous similar messages
[ 6025.187438] Lustre: 3648:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789055917/real 1789055917]  req@ffff8a695f67bb80 x1875954918476928/t0(0) o400->MGC192.168.204.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789055933 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6025.213232] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6031.095114] LDISKFS-fs (dm-0): 3 truncates cleaned up
[ 6031.099098] LDISKFS-fs (dm-0): recovery complete
[ 6031.106447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 6034.414439] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x12bf7cdc4022cac9
[ 6034.928570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6035.019206] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 6035.341918] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 6039.746146] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6040.145929] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted.
[ 6040.226668] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:611 to 0x280000401:641)
[ 6040.227654] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:504 to 0x2c0000401:577)
[ 6051.648875] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6053.593446] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6059.543514] Lustre: DEBUG MARKER: (dd_pid=131875, time=0, timeout=600)
[ 6102.795706] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 12:00:09 (1789056009)
[ 6155.928516] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 12:01:02 (1789056062)
[ 6190.126668] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 6203.946918] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 6206.016102] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 6207.864609] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 6210.084021] Lustre: DEBUG MARKER: Set quota for 1 times
[ 6213.908246] Lustre: DEBUG MARKER: Set quota for 2 times
[ 6217.622889] Lustre: DEBUG MARKER: Set quota for 3 times
[ 6221.258551] Lustre: DEBUG MARKER: Set quota for 4 times
[ 6225.087197] Lustre: DEBUG MARKER: Set quota for 5 times
[ 6228.661230] Lustre: DEBUG MARKER: Set quota for 6 times
[ 6232.824424] Lustre: DEBUG MARKER: Set quota for 7 times
[ 6237.102792] Lustre: DEBUG MARKER: Set quota for 8 times
[ 6272.799470] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 12:02:59 (1789056179)
[ 6290.912349] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 6290.920989] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6290.926505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6290.926515] Lustre: Skipped 4 previous similar messages
[ 6290.966451] Lustre: Skipped 8 previous similar messages
[ 6294.553824] Lustre: server umount lustre-MDT0000 complete
[ 6296.045516] LustreError: 7806:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6296.071471] LustreError: 7806:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 34 previous similar messages
[ 6298.599126] LustreError: 10345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789056207 with bad export cookie 1350935698645109449
[ 6298.612704] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6298.622506] LustreError: 10345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 6299.427403] Lustre: server umount lustre-MDT0001 complete
[ 6316.248088] Lustre: server umount lustre-OST0000 complete
[ 6316.320478] Lustre: 3650:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789056209/real 1789056209]  req@ffff8a697cde0380 x1875954918669696/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789056225 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6318.371875] Lustre: 3648:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789056211/real 1789056211]  req@ffff8a684e38b800 x1875954918670080/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789056227 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6320.442484] Lustre: server umount lustre-OST0001 complete
[ 6339.092536] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local
[ 6350.705537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 6351.591255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6357.037972] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6369.479085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 6370.076827] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 6375.071854] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6378.466664] Lustre: 156241:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6385.329642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 6385.706736] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 6393.125638] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6394.930221] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97)
[ 6396.988862] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:643 to 0x280000401:673)
[ 6404.487964] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 6404.627718] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6409.705263] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:580 to 0x2c0000401:609)
[ 6412.796299] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6422.385467] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6426.985795] Lustre: 158121:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6450.667123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6450.678541] Lustre: Skipped 6 previous similar messages
[ 6455.779663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6455.793593] Lustre: Skipped 3 previous similar messages
[ 6456.632708] Lustre: server umount lustre-MDT0000 complete
[ 6461.041488] LustreError: 155071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789056369 with bad export cookie 1350935698645119767
[ 6461.059370] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6461.486647] Lustre: server umount lustre-MDT0001 complete
[ 6476.854176] Lustre: server umount lustre-OST0000 complete
[ 6481.378780] Lustre: 3648:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789056374/real 1789056374]  req@ffff8a684790aa00 x1875954918757888/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789056390 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6482.085216] Lustre: server umount lustre-OST0001 complete
[ 6503.737715] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local
[ 6516.162874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 6516.820267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6522.133353] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6532.410124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 6537.950426] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6541.141809] Lustre: 161859:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6548.957137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 6549.482384] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 6549.497536] Lustre: Skipped 1 previous similar message
[ 6556.516617] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6557.678507] LustreError: 162213:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6557.696014] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:129)
[ 6557.697010] LustreError: 162213:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 17 previous similar messages
[ 6561.774336] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:643 to 0x280000401:705)
[ 6565.271846] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 6570.999853] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:580 to 0x2c0000401:641)
[ 6573.194761] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6582.735359] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6587.710654] Lustre: 163729:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6600.411484] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 12:08:26 (1789056506)
[ 6602.662166] Lustre: DEBUG MARKER: OST0_SIZE: 3605396 required: 6144
[ 6604.794722] Lustre: DEBUG MARKER: run for 4MB test file
[ 6619.126307] Lustre: DEBUG MARKER: User quota (limit: 4 MB)
[ 6626.819792] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT
[ 6629.036684] Lustre: DEBUG MARKER: Write half of file
[ 6631.302578] Lustre: DEBUG MARKER: Write out of block quota ...
[ 6633.554929] Lustre: DEBUG MARKER: Step1: done
[ 6635.741933] Lustre: DEBUG MARKER: Step2: rewrite should succeed
[ 6637.832805] Lustre: DEBUG MARKER: Step2: done
[ 6667.870171] Lustre: DEBUG MARKER: OST0_SIZE: 3605396 required: 61440
[ 6669.619496] Lustre: DEBUG MARKER: run for 40MB test file
[ 6685.927794] Lustre: DEBUG MARKER: User quota (limit: 40 MB)
[ 6694.972079] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT
[ 6697.881484] Lustre: DEBUG MARKER: Write half of file
[ 6702.264210] Lustre: DEBUG MARKER: Write out of block quota ...
[ 6707.024249] Lustre: DEBUG MARKER: Step1: done
[ 6709.293315] Lustre: DEBUG MARKER: Step2: rewrite should succeed
[ 6712.488184] Lustre: DEBUG MARKER: Step2: done
[ 6778.247561] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 12:11:24 (1789056684)
[ 6830.945449] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 12:12:17 (1789056737)
[ 6877.665853] Lustre: DEBUG MARKER: Write...
[ 6880.911376] Lustre: DEBUG MARKER: Write out of block quota ...
[ 6938.123385] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 12:14:04 (1789056844)
[ 6945.974642] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 12:14:12 (1789056852)
[ 6957.066993] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 12:14:23 (1789056863)
[ 6966.329641] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 12:14:33 (1789056873)
[ 6976.242585] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 12:14:42 (1789056882)
[ 7035.553075] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 7187.367711] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 7351.246325] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 12:20:57 (1789057257)
[ 7405.722473] Lustre: DEBUG MARKER: Restart...
[ 7410.658906] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 7410.679930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 7410.681371] Lustre: Skipped 9 previous similar messages
[ 7410.702853] Lustre: Skipped 3 previous similar messages
[ 7415.777357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 7415.792851] Lustre: Skipped 3 previous similar messages
[ 7416.836135] Lustre: server umount lustre-MDT0000 complete
[ 7420.900617] LustreError: 160716:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 7420.930643] LustreError: 160716:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages
[ 7421.278678] LustreError: 160701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789057329 with bad export cookie 1350935698645122070
[ 7421.286713] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 7421.299839] LustreError: 160701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages
[ 7421.825916] Lustre: server umount lustre-MDT0001 complete
[ 7436.954609] Lustre: server umount lustre-OST0000 complete
[ 7450.508354] Lustre: server umount lustre-OST0001 complete
[ 7470.696766] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local
[ 7484.389477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 7485.281112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 7485.298077] Lustre: Skipped 1 previous similar message
[ 7491.500429] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7502.146473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 7502.422309] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 7507.496888] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7511.200625] Lustre: 187720:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 7517.968494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 7525.853917] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7527.635502] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:161)
[ 7529.743691] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:716 to 0x280000401:737)
[ 7535.482922] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 7535.789538] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 7535.793741] Lustre: Skipped 1 previous similar message
[ 7541.247792] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:650 to 0x2c0000401:673)
[ 7542.217775] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7551.902335] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 7557.167923] Lustre: 189592:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 7640.713216] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 12:25:47 (1789057547)
[ 7700.776645] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 12:26:47 (1789057607)
[ 9628.763732] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 12:58:55 (1789059535)
[ 9640.424694] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 9640.427388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 9640.452045] Lustre: Skipped 4 previous similar messages
[ 9640.462500] Lustre: Skipped 2 previous similar messages
[ 9645.537275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 9645.544542] Lustre: Skipped 4 previous similar messages
[ 9645.819583] Lustre: server umount lustre-MDT0000 complete
[ 9650.298380] LustreError: 186558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789059559 with bad export cookie 1350935698645131184
[ 9650.307615] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 9650.313610] LustreError: 186558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 9650.657842] LustreError: 186574:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9650.688328] LustreError: 186574:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 12 previous similar messages
[ 9651.211632] Lustre: server umount lustre-MDT0001 complete
[ 9656.436216] Lustre: server umount lustre-OST0000 complete
[ 9661.898478] Lustre: server umount lustre-OST0001 complete
[ 9683.180142] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local
[ 9696.332139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 9697.349312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 9703.456650] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9715.085372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 9715.949513] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 9722.594708] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9725.952764] Lustre: 198256:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 9734.950837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 9735.422860] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 9740.518216] LustreError: 198611:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9740.529517] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:193)
[ 9740.542453] LustreError: 198611:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages
[ 9742.576163] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5740 to 0x280000401:5761)
[ 9742.900545] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9754.812825] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 9755.183452] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 9760.244170] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5674 to 0x2c0000401:5697)
[ 9762.407423] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9770.970799] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 9775.261529] Lustre: 200140:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 9805.592555] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 13:01:51 (1789059711)
[ 9838.691188] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 13:02:24 (1789059744)
[ 9867.428592] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 13:02:54 (1789059774)
[ 9906.126736] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 13:03:32 (1789059812)
[ 9947.108612] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 13:04:13 (1789059853)
[10035.282213] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 13:05:41 (1789059941)
[10078.147623] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 13:06:24 (1789059984)
[10151.703332] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 13:07:38 (1789060058)
[10159.051744] Lustre: *** cfs_fail_loc=a09, val=0***
[10159.056558] Lustre: Skipped 2 previous similar messages
[10161.079391] Lustre: *** cfs_fail_loc=a09, val=0***
[10161.084540] Lustre: Skipped 55 previous similar messages
[10165.124264] Lustre: *** cfs_fail_loc=a09, val=0***
[10165.127077] Lustre: Skipped 127 previous similar messages
[10173.154931] Lustre: *** cfs_fail_loc=a09, val=0***
[10173.156922] Lustre: Skipped 305 previous similar messages
[10189.166678] Lustre: *** cfs_fail_loc=a09, val=0***
[10189.170591] Lustre: Skipped 595 previous similar messages
[10221.224813] Lustre: *** cfs_fail_loc=a09, val=0***
[10221.229304] Lustre: Skipped 1087 previous similar messages
[10285.298056] Lustre: *** cfs_fail_loc=a09, val=0***
[10285.306550] Lustre: Skipped 2219 previous similar messages
[10809.825073] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[10809.825948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[10809.856228] Lustre: Skipped 4 previous similar messages
[10809.869900] Lustre: Skipped 1 previous similar message
[10814.832081] Lustre: server umount lustre-MDT0000 complete
[10814.947516] LustreError: 197108:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[10814.965026] LustreError: 197108:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages
[10818.946561] LustreError: 197095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789060727 with bad export cookie 1350935698646913195
[10818.951383] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[10818.958315] LustreError: 197095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages
[10820.064742] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[10820.084459] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping)
[10820.085558] Lustre: Skipped 2 previous similar messages
[10820.115551] Lustre: Skipped 3 previous similar messages
[10823.074258] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping)
[10823.085268] Lustre: Skipped 1 previous similar message
[10828.257597] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping)
[10833.380440] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck?
[10833.850945] Lustre: server umount lustre-MDT0001 complete
[10838.911835] Lustre: server umount lustre-OST0000 complete
[10844.302158] Lustre: server umount lustre-OST0001 complete
[10852.711872] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid
[10863.335599] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local
[10919.743226] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local
[10934.167346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[10934.441312] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000
[10934.478968] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[10934.545297] Lustre: lustre-MDT0000: new disk, initializing
[10934.598339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[10934.611506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[10940.471324] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10952.760947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[10952.829731] Lustre: 216442:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17
[10952.861767] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
[10952.864804] Lustre: Skipped 1 previous similar message
[10952.936150] Lustre: lustre-MDT0001: new disk, initializing
[10953.001551] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[10953.053372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt
[10953.058158] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt]
[10959.043723] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10963.991373] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params
[10970.949141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[10971.171173] Lustre: lustre-OST0000: new disk, initializing
[10971.174032] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[10971.178559] Lustre: 218076:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0]
[10971.235241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[10972.516847] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost
[10972.543386] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost]
[10972.687973] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401
[10978.261794] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10988.589823] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[10988.817823] Lustre: lustre-OST0001: new disk, initializing
[10988.823357] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[10988.833967] Lustre: 218949:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0]
[10988.891427] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[10989.902164] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost
[10989.923536] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost]
[10990.062085] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401
[10995.637923] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[11006.017577] Lustre: DEBUG MARKER: Using TIMEOUT=20
[11009.557197] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params
[11037.555230] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 13:22:24 (1789060944)
[11044.195659] Lustre: DEBUG MARKER: == sanity-quota test 49c: lfs quota long options don't consume an extra argument ========================================================== 13:22:31 (1789060951)
[11052.753404] Lustre: DEBUG MARKER: == sanity-quota test 49d: lfs quota -d and --delimiter both work ========================================================== 13:22:39 (1789060959)
[11060.004329] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 13:22:46 (1789060966)
[11092.318883] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 13:23:19 (1789060999)
[11146.156734] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 13:24:12 (1789061052)
[11171.952298] Lustre: DEBUG MARKER: rename directory return 255
[11211.616244] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 13:25:18 (1789061118)
[11234.073778] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 13:25:40 (1789061140)
[11275.100872] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 13:26:21 (1789061181)
[11416.027941] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 13:28:42 (1789061322)
[11444.668755] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 13:29:11 (1789061351)
[11484.144512] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 13:29:51 (1789061391)
[11641.129722] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 13:32:27 (1789061547)
[11645.414162] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[11645.415718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[11645.444081] Lustre: Skipped 1 previous similar message
[11645.449607] Lustre: Skipped 3 previous similar messages
[11650.534754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[11650.545593] Lustre: Skipped 3 previous similar messages
[11651.558199] Lustre: server umount lustre-MDT0000 complete
[11655.662163] LustreError: 220700:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[11655.699120] LustreError: 220700:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages
[11656.312985] LustreError: 216434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789061565 with bad export cookie 1350935698647285595
[11656.342666] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[11656.846650] Lustre: server umount lustre-MDT0001 complete
[11672.404692] Lustre: server umount lustre-OST0000 complete
[11682.783608] LustreError: 3651:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED  req@ffff8a6842388a80 x1875954930101760/t0(0) o601->lustre-MDT0000-lwp-OST0001@0@lo:23/10 lens 336/336 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[11682.814842] LustreError: 3651:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8  qsd:lustre-OST0001 qtype:grp lqe: ffff8a6848fbb980 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 1516 qunit:0 qtune:0 edquot:0 default:no revoke:0
[11682.851411] LustreError: 3651:0:(qsd_handler.c:298:qsd_req_completion()) Skipped 1 previous similar message
[11687.045380] Lustre: server umount lustre-OST0001 complete
[11710.556580] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local
[11722.951579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[11723.376025] LustreError: 236583:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[11723.442465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[11723.463920] LustreError: lustre-MDT0000: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[11727.737648] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[11728.884099] LustreError: 236584:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[11739.271085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[11739.744626] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[11739.761715] LustreError: lustre-MDT0001: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[11746.832957] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[11750.564388] Lustre: 237730:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[11759.221098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[11759.813071] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[11759.833145] LustreError: lustre-OST0000: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[11764.899225] LustreError: 238082:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[11764.944831] LustreError: 238082:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages
[11767.936391] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[11770.048548] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:20 to 0x280000401:65)
[11778.124800] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[11778.367335] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[11778.377355] LustreError: lustre-OST0001: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[11783.703786] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:20 to 0x2c0000401:65)
[11785.989839] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[11794.751982] Lustre: DEBUG MARKER: Using TIMEOUT=20
[11798.695743] Lustre: 239603:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[11807.114108] LustreError: 236578:0:(osd_handler.c:3421:osd_quota_transfer()) lustre-MDT0000: quota transfer failed. Is project enforcement enabled on the ldiskfs filesystem? rc = -95
[11814.376155] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[11814.381115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[11814.391059] Lustre: Skipped 5 previous similar messages
[11814.402867] Lustre: Skipped 6 previous similar messages
[11818.231624] Lustre: server umount lustre-MDT0000 complete
[11819.521470] LustreError: 236583:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[11819.570458] LustreError: 236583:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages
[11823.297071] LustreError: 236564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789061732 with bad export cookie 1350935698647334490
[11823.301962] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[11823.310971] LustreError: 236564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages
[11823.810796] Lustre: server umount lustre-MDT0001 complete
[11838.351932] Lustre: server umount lustre-OST0000 complete
[11840.480160] Lustre: 3651:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789061733/real 1789061733]  req@ffff8a684a5cb100 x1875954930158464/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789061749 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[11840.497938] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[11843.066941] Lustre: server umount lustre-OST0001 complete
[11867.715022] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local
[11879.003322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[11879.635067] LustreError: 241993:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[11879.710799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[11884.725452] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[11894.873886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[11900.289655] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[11903.254100] Lustre: 243141:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[11910.324318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[11918.196510] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[11924.977887] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:20 to 0x280000401:97)
[11929.597299] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[11935.230101] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97)
[11937.344651] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[11947.357622] Lustre: DEBUG MARKER: Using TIMEOUT=20
[11951.375740] Lustre: 245016:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[11985.446476] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 13:38:11 (1789061891)
[12041.117967] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[12044.295983] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 13:39:10 (1789061950)
[12070.551743] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[12073.028183] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 13:39:39 (1789061979)
[12121.378743] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[12123.523034] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 13:40:29 (1789062029)
[12170.414213] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 13:41:17 (1789062077)
[12183.661422] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[12191.099813] Lustre: 251587:0:(qsd_reint.c:249:qsd_reint_index()) lustre-OST0000: index version for fid [0x200000005:0x1007:0x0] is 0, but index isn't empty (1)
[12191.123899] Lustre: 251587:0:(qsd_reint.c:249:qsd_reint_index()) Skipped 1 previous similar message
[12196.531515] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[12203.232733] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[12207.472474] Lustre: DEBUG MARKER: Write...
[12230.749419] LustreError: 252934:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[12237.582922] Lustre: DEBUG MARKER: Write...
[12253.054384] Lustre: DEBUG MARKER: Write...
[12333.238603] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 13:43:59 (1789062239)
[12361.804224] LustreError: 256902:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[12413.040381] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 13:45:19 (1789062319)
[12447.402139] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[12449.492262] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[12547.100082] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 13:47:33 (1789062453)
[12594.731732] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 13:48:21 (1789062501)
[12636.259902] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 13:49:02 (1789062542)
[12650.238949] Lustre: DEBUG MARKER: User quota (block hardlimit:100 MB)
[12678.398447] Lustre: DEBUG MARKER: Write...
[12680.874448] Lustre: DEBUG MARKER: Write out of block quota ...
[12777.550846] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 13:51:24 (1789062684)
[12792.956928] Lustre: DEBUG MARKER: User quota (block hardlimit:1000 MB)
[12823.292726] LustreError: 246200:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool2 id:60000 enforced:1 hard:10240 soft:0 granted:10240 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[12886.678185] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 13:53:12 (1789062792)
[12899.867785] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB)
[12920.258047] Lustre: DEBUG MARKER: Write...
[12922.694728] Lustre: DEBUG MARKER: Write out of block quota ...
[12984.846976] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 13:54:50 (1789062890)
[13017.593535] Lustre: DEBUG MARKER: set to use default quota
[13020.107921] Lustre: DEBUG MARKER: set default quota
[13022.877870] Lustre: DEBUG MARKER: get default quota
[13033.773128] Lustre: DEBUG MARKER: Test not out of quota
[13041.878196] Lustre: DEBUG MARKER: Test out of quota
[13062.578207] Lustre: DEBUG MARKER: Increase default quota
[13091.689484] Lustre: DEBUG MARKER: Set quota to override default quota
[13091.774186] LustreError: 258488:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45056 time:1789667800 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[13111.905711] Lustre: DEBUG MARKER: Set to use default quota again
[13136.047674] Lustre: DEBUG MARKER: Cleanup
[13206.874391] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 13:58:33 (1789063113)
[13234.607455] Lustre: DEBUG MARKER: set default quota for qpool1
[13236.670580] Lustre: DEBUG MARKER: Write from user that hasn't lqe
[13294.557309] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 14:00:00 (1789063200)
[13386.464262] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 14:01:32 (1789063292)
[13460.218774] Lustre: DEBUG MARKER: Write...
[13463.319917] Lustre: DEBUG MARKER: Write out of block quota ...
[13581.359154] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 14:04:47 (1789063487)
[13630.565579] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 14:05:36 (1789063536)
[13640.496179] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 14:05:46 (1789063546)
[13689.613401] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 14:06:35 (1789063595)
[13738.477645] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 14:07:25 (1789063645)
[13764.827791] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 14:07:51 (1789063671)
[13804.974608] Lustre: *** cfs_fail_loc=a06, val=0***
[13804.978147] Lustre: Skipped 3605 previous similar messages
[13805.476159] LustreError: 242007:0:(qmt_lock.c:479:qmt_lvbo_update()) $$$ failed to release quota space on glimpse 0!=2048 : rc = -11
                 qmt:lustre-QMT0000 pool:dt-0x0 id:60000 enforced:1 hard:102400 soft:0 granted:25600 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no
[13811.040814] Lustre: Failing over lustre-OST0001
[13811.289632] Lustre: server umount lustre-OST0001 complete
[13814.245240] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107
[13814.245404] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[13814.251292] LustreError: Skipped 1 previous similar message
[13814.257872] LustreError: 243494:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[13814.267389] Lustre: Skipped 1 previous similar message
[13814.304964] LustreError: 243494:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages
[13822.783315] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[13823.136665] Lustre: lustre-OST0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180
[13823.162240] Lustre: lustre-OST0001: in recovery but waiting for the first client to connect
[13824.741871] Lustre: lustre-OST0001: Will be in recovery for at least 1:00, or until 3 clients reconnect
[13824.996824] Lustre: lustre-OST0001: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted.
[13825.004507] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo)
[13825.019308] Lustre: Skipped 8 previous similar messages
[13831.564930] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[13883.806676] Lustre: DEBUG MARKER: == sanity-quota test 81: Race qmt_start_pool_recalc with qmt_pool_free ========================================================== 14:09:50 (1789063790)
[13897.732788] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[13906.801063] LustreError: 295846:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 sleeping for 10000ms
[13912.544373] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[13912.555511] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[13912.596724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[13915.617168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[13915.619642] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[13915.640973] Lustre: Skipped 3 previous similar messages
[13915.670470] Lustre: Skipped 2 previous similar messages
[13916.824191] LustreError: 295846:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 awake
[13917.132929] Lustre: server umount lustre-MDT0000 complete
[13918.073326] LustreError: 241989:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[13918.099541] LustreError: 241989:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages
[13927.519980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[13927.672565] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[13928.129370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[13928.135453] Lustre: Skipped 3 previous similar messages
[13928.199630] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:112 to 0x280000401:129)
[13928.202897] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:113 to 0x2c0000401:129)
[13933.546366] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo)
[13933.555025] Lustre: Skipped 1 previous similar message
[13934.515428] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[13973.455236] Lustre: DEBUG MARKER: == sanity-quota test 82: verify more than 8 qids for single operation ========================================================== 14:11:19 (1789063879)
[14004.954633] Lustre: DEBUG MARKER: == sanity-quota test 83: Setting default quota shouldn't affect grace time ========================================================== 14:11:51 (1789063911)
[14035.842373] Lustre: DEBUG MARKER: == sanity-quota test 84: Reset quota should fix the insane granted quota ========================================================== 14:12:21 (1789063941)
[14092.752255] Lustre: *** cfs_fail_loc=a08, val=0***
[14092.756103] Lustre: Skipped 2681 previous similar messages
[14092.766730] Lustre: *** cfs_fail_loc=a08, val=0***
[14092.778667] Lustre: Skipped 3 previous similar messages
[14193.548509] Lustre: DEBUG MARKER: == sanity-quota test 85: do not hung at write with the least_qunit ========================================================== 14:14:59 (1789064099)
[14297.151179] Lustre: DEBUG MARKER: == sanity-quota test 86: Pre-acquired quota should be released if quota is over limit ========================================================== 14:16:43 (1789064203)
[14390.780478] LustreError: 241989:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:16384 time:1789669099 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[14562.244366] LustreError: 258488:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:16384 time:1789669270 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[14562.301451] LustreError: 3648:0:(qsd_reint.c:633:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:1 until pending updates are flushed.
[14572.434255] Lustre: DEBUG MARKER: sanity-quota test_86: @@@@@@ FAIL: succeed to create files, expect failed