[ 1545.752641] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 1547.584650] Lustre: DEBUG MARKER: Files for user (quota_usr), count=0:
[ 1548.136236] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:27776).
[ 1548.671328] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 1551.163834] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 1551.666805] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 1552.296542] Lustre: Failing over lustre-MDT0000
[ 1552.352778] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 1552.356890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 1552.645748] Lustre: server umount lustre-MDT0000 complete
[ 1553.117302] LustreError: lustre-MDT0000-lwp-OST0000: operation quota_acquire to node 0@lo failed: rc = -107
[ 1553.119941] LustreError: 6488:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 1553.125094] LustreError: 6488:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages
[ 1557.472522] LustreError: 6489:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 1557.478762] LustreError: 6489:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages
[ 1566.558489] LDISKFS-fs (dm-0): 10 truncates cleaned up
[ 1566.559611] LDISKFS-fs (dm-0): recovery complete
[ 1566.564070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 1566.600799] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 1566.631771] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.44@tcp (not set up)
[ 1566.634190] Lustre: Skipped 2 previous similar messages
[ 1566.669243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 1566.707012] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 1567.942693] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1568.362608] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 1571.810658] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 1571.813096] Lustre: Skipped 1 previous similar message
[ 1571.823057] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted.
[ 1571.838612] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1801 to 0x280000401:1825)
[ 1571.838621] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1760 to 0x2c0000401:1793)
[ 1573.520331] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 1574.018331] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1583.579441] Lustre: DEBUG MARKER: (dd_pid=100971, time=7, timeout=600)
[ 1598.987406] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 1600.721920] Lustre: DEBUG MARKER: Files for user (quota_usr), count=0:
[ 1601.251321] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:95232).
[ 1601.805250] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 1604.441582] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 1604.980325] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 1605.667139] Lustre: Failing over lustre-MDT0000
[ 1605.909337] Lustre: server umount lustre-MDT0000 complete
[ 1606.624449] LustreError: lustre-MDT0000-lwp-OST0001: operation quota_acquire to node 0@lo failed: rc = -107
[ 1606.624493] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 1606.627677] LustreError: Skipped 1 previous similar message
[ 1606.627956] LustreError: 7655:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 1606.632095] Lustre: Skipped 4 previous similar messages
[ 1606.640866] LustreError: 7655:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages
[ 1619.902718] LDISKFS-fs (dm-0): 2 truncates cleaned up
[ 1619.904040] LDISKFS-fs (dm-0): recovery complete
[ 1619.906598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 1619.942069] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 1620.019510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 1620.056059] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 1621.352756] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1623.976487] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 1625.059372] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[ 1625.062063] Lustre: Skipped 3 previous similar messages
[ 1625.070288] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted.
[ 1625.086593] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1827 to 0x280000401:1857)
[ 1625.086682] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1760 to 0x2c0000401:1825)
[ 1626.944838] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 1627.482295] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1629.817085] Lustre: DEBUG MARKER: (dd_pid=103867, time=0, timeout=600)
[ 1648.651885] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 04:42:55 (1789893775)
[ 1654.875856] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 1655.423237] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:102400).
[ 1657.329541] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 1657.876932] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:102400).
[ 1669.981772] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 04:43:16 (1789893796)
[ 1678.707861] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 1682.095254] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 1682.559546] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 1683.026181] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 1683.576562] Lustre: DEBUG MARKER: Set quota for 1 times
[ 1685.250878] Lustre: DEBUG MARKER: Set quota for 2 times
[ 1686.911967] Lustre: DEBUG MARKER: Set quota for 3 times
[ 1688.497065] Lustre: DEBUG MARKER: Set quota for 4 times
[ 1690.055903] Lustre: DEBUG MARKER: Set quota for 5 times
[ 1691.654590] Lustre: DEBUG MARKER: Set quota for 6 times
[ 1693.252035] Lustre: DEBUG MARKER: Set quota for 7 times
[ 1694.850770] Lustre: DEBUG MARKER: Set quota for 8 times
[ 1696.458628] Lustre: DEBUG MARKER: Set quota for 9 times
[ 1697.996742] Lustre: DEBUG MARKER: Set quota for 10 times
[ 1699.579546] Lustre: DEBUG MARKER: Set quota for 11 times
[ 1701.226329] Lustre: DEBUG MARKER: Set quota for 12 times
[ 1703.038093] Lustre: DEBUG MARKER: Set quota for 13 times
[ 1704.680695] Lustre: DEBUG MARKER: Set quota for 14 times
[ 1706.372369] Lustre: DEBUG MARKER: Set quota for 15 times
[ 1708.076371] Lustre: DEBUG MARKER: Set quota for 16 times
[ 1709.722806] Lustre: DEBUG MARKER: Set quota for 17 times
[ 1711.348189] Lustre: DEBUG MARKER: Set quota for 18 times
[ 1727.020214] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 04:44:13 (1789893853)
[ 1735.635963] Lustre: server umount lustre-MDT0000 complete
[ 1736.739629] LustreError: 74266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789893864 with bad export cookie 4697206803269881629
[ 1736.741494] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 1736.743125] LustreError: 74266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 1736.844413] Lustre: server umount lustre-MDT0001 complete
[ 1753.312111] Lustre: 3616:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789893865/real 1789893865]  req@ffff91aa43763800 x1876837994754176/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789893881 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 1753.320265] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 1753.324531] Lustre: Skipped 2 previous similar messages
[ 1757.408133] Lustre: 3616:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789893869/real 1789893869]  req@ffff91a954a02300 x1876837994754816/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789893885 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 1757.416145] Lustre: 3616:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 1757.698764] Lustre: server umount lustre-OST0000 complete
[ 1758.919302] Lustre: server umount lustre-OST0001 complete
[ 1763.649827] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local
[ 1766.854024] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 1766.970558] LustreError: 115619:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 1766.977178] LustreError: 115619:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages
[ 1766.991469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 1768.164983] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1770.740075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 1770.847743] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 1772.045185] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1772.938700] Lustre: 116774:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 1774.888124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 1774.978402] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 1776.741840] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1777.058856] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1859 to 0x280000401:1889)
[ 1779.339921] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 1780.453609] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97)
[ 1780.453921] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1828 to 0x2c0000401:1857)
[ 1781.299976] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1784.196167] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 1785.235703] Lustre: 118647:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 1791.504470] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 1791.506414] LustreError: Skipped 1 previous similar message
[ 1791.507517] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 1791.510306] Lustre: Skipped 1 previous similar message
[ 1791.511451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 1796.065335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 1796.067271] Lustre: Skipped 3 previous similar messages
[ 1797.717054] Lustre: server umount lustre-MDT0000 complete
[ 1798.912781] LustreError: 115601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789893926 with bad export cookie 4697206803269891737
[ 1798.914684] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 1798.916339] LustreError: 115601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 1799.021970] Lustre: server umount lustre-MDT0001 complete
[ 1810.369753] Lustre: server umount lustre-OST0000 complete
[ 1821.780471] Lustre: server umount lustre-OST0001 complete
[ 1826.966583] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local
[ 1830.384492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 1830.515209] LustreError: 121287:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 1830.520430] LustreError: 121287:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message
[ 1830.541339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 1830.543551] Lustre: Skipped 1 previous similar message
[ 1831.800444] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1834.399284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 1835.783054] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1836.704033] Lustre: 122428:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 1838.880538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 1840.807656] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1841.058997] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1859 to 0x280000401:1921)
[ 1843.465690] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 1844.579018] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:129)
[ 1844.579561] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1828 to 0x2c0000401:1889)
[ 1845.244969] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1848.065684] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 1849.097691] Lustre: 124299:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 1852.099221] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 04:46:18 (1789893978)
[ 1852.568893] Lustre: DEBUG MARKER: OST0_SIZE: 3605412 required: 6144
[ 1852.994409] Lustre: DEBUG MARKER: run for 4MB test file
[ 1857.492462] Lustre: DEBUG MARKER: User quota (limit: 4 MB)
[ 1859.174853] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT
[ 1859.633790] Lustre: DEBUG MARKER: Write half of file
[ 1860.171132] Lustre: DEBUG MARKER: Write out of block quota ...
[ 1860.734775] Lustre: DEBUG MARKER: Step1: done
[ 1861.218578] Lustre: DEBUG MARKER: Step2: rewrite should succeed
[ 1861.719309] Lustre: DEBUG MARKER: Step2: done
[ 1875.570540] Lustre: DEBUG MARKER: OST0_SIZE: 3605412 required: 61440
[ 1876.044900] Lustre: DEBUG MARKER: run for 40MB test file
[ 1879.395426] Lustre: DEBUG MARKER: User quota (limit: 40 MB)
[ 1881.120335] Lustre: DEBUG MARKER: Files for user (quota_usr), count=0:
[ 1881.586907] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2048).
[ 1882.085414] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT
[ 1882.546854] Lustre: DEBUG MARKER: Write half of file
[ 1883.373519] Lustre: DEBUG MARKER: Write out of block quota ...
[ 1884.119617] Lustre: DEBUG MARKER: Step1: done
[ 1884.605795] Lustre: DEBUG MARKER: Step2: rewrite should succeed
[ 1885.115597] Lustre: DEBUG MARKER: Step2: done
[ 1905.740567] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 04:47:12 (1789894032)
[ 1927.441642] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 04:47:33 (1789894053)
[ 1944.665137] Lustre: DEBUG MARKER: sanity-quota test_25: @@@@@@ FAIL: Used space(6144) for user quota_usr isn't 0.