[ 1642.074206] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 1649.668400] Lustre: DEBUG MARKER: Files for user (quota_usr), count=0:
[ 1651.920888] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:27772).
[ 1654.195948] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 1665.566839] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 1667.656325] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 1670.317995] Lustre: Failing over lustre-MDT0000
[ 1670.901583] Lustre: server umount lustre-MDT0000 complete
[ 1671.098245] LustreError: lustre-MDT0000-lwp-OST0000: operation quota_acquire to node 0@lo failed: rc = -107
[ 1671.116512] LustreError: Skipped 1 previous similar message
[ 1671.124542] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 1671.152251] Lustre: Skipped 1 previous similar message
[ 1671.154205] LustreError: 7778:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 1671.213812] LustreError: 7778:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages
[ 1676.379531] LustreError: 10070:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 1676.437208] LustreError: 10070:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages
[ 1684.969793] LustreError: 6488:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 1685.011981] LustreError: 6488:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages
[ 1691.168743] Lustre: 3621:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789890849/real 1789890849]  req@ffff9069b5681c00 x1876834907201024/t0(0) o400->MGC192.168.204.133@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789890865 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 1691.217429] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 1701.866445] LustreError: 6487:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 1701.888843] LustreError: 6487:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 19 previous similar messages
[ 1703.139385] LDISKFS-fs (dm-0): 10 truncates cleaned up
[ 1703.141801] LDISKFS-fs (dm-0): recovery complete
[ 1703.152851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 1716.192980] LustreError: 3618:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight  req@ffff906889e07b80 x1876834907214976/t0(0) o250->MGC192.168.204.133@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 1716.507359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 1716.545826] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 1717.731735] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 1719.649467] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1721.828981] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 1721.838060] Lustre: Skipped 1 previous similar message
[ 1721.864082] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted.
[ 1721.906903] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1609 to 0x280000401:1633)
[ 1721.907530] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1590 to 0x2c0000401:1633)
[ 1726.996455] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 1728.340193] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1740.619794] Lustre: DEBUG MARKER: (dd_pid=101509, time=8, timeout=600)
[ 1758.734204] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 1762.472297] Lustre: DEBUG MARKER: Files for user (quota_usr), count=0:
[ 1763.576911] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:99328).
[ 1764.726805] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 1770.042147] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 1771.183238] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 1772.498761] Lustre: Failing over lustre-MDT0000
[ 1772.872294] Lustre: server umount lustre-MDT0000 complete
[ 1773.024453] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 1773.027336] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 1773.034752] LustreError: Skipped 2 previous similar messages
[ 1773.044803] Lustre: Skipped 2 previous similar messages
[ 1773.048524] LustreError: 10070:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 1773.071325] LustreError: 10070:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages
[ 1788.321136] Lustre: 3622:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789890947/real 1789890947]  req@ffff90688e0ace00 x1876834907264768/t0(0) o400->MGC192.168.204.133@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789890963 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 1788.347724] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 1791.154171] LDISKFS-fs (dm-0): 2 truncates cleaned up
[ 1791.156400] LDISKFS-fs (dm-0): recovery complete
[ 1791.161507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 1798.626645] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x5c4f93dd001a26a8
[ 1798.634844] Lustre: MGC192.168.204.133@tcp: Connection restored to 0@lo (at 0@lo)
[ 1798.639649] Lustre: Skipped 3 previous similar messages
[ 1798.766295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 1798.799209] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 1799.244834] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 1801.583540] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1804.262944] LustreError: 3618:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-lwp-MDT0001: namespace resource [0x200000006:0x10000:0x0].0x0 (ffff906984f3df00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 1804.296385] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted.
[ 1804.329728] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1590 to 0x2c0000401:1665)
[ 1804.330287] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1635 to 0x280000401:1665)
[ 1807.709791] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 1808.599688] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1812.573202] Lustre: DEBUG MARKER: (dd_pid=104301, time=0, timeout=600)
[ 1836.318624] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 03:56:49 (1789891009)
[ 1848.490234] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 1849.612732] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:102400).
[ 1853.460517] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 1854.637972] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:102400).
[ 1874.260879] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 03:57:27 (1789891047)
[ 1891.420921] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 1898.831774] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 1899.864131] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 1900.966629] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 1902.119902] Lustre: DEBUG MARKER: Set quota for 1 times
[ 1904.469805] Lustre: DEBUG MARKER: Set quota for 2 times
[ 1906.718488] Lustre: DEBUG MARKER: Set quota for 3 times
[ 1909.100502] Lustre: DEBUG MARKER: Set quota for 4 times
[ 1911.352884] Lustre: DEBUG MARKER: Set quota for 5 times
[ 1913.281811] Lustre: DEBUG MARKER: Set quota for 6 times
[ 1915.531543] Lustre: DEBUG MARKER: Set quota for 7 times
[ 1917.579305] Lustre: DEBUG MARKER: Set quota for 8 times
[ 1919.786186] Lustre: DEBUG MARKER: Set quota for 9 times
[ 1921.890819] Lustre: DEBUG MARKER: Set quota for 10 times
[ 1923.987602] Lustre: DEBUG MARKER: Set quota for 11 times
[ 1926.315196] Lustre: DEBUG MARKER: Set quota for 12 times
[ 1928.473124] Lustre: DEBUG MARKER: Set quota for 13 times
[ 1930.828814] Lustre: DEBUG MARKER: Set quota for 14 times
[ 1948.156476] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 03:58:42 (1789891122)
[ 1956.117658] Lustre: server umount lustre-MDT0000 complete
[ 1957.346772] LustreError: 8421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789891132 with bad export cookie 6651697752048281256
[ 1957.348627] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 1957.351105] LustreError: 8421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 1957.449875] Lustre: server umount lustre-MDT0001 complete
[ 1974.111121] Lustre: 3619:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789891132/real 1789891132]  req@ffff906990ece680 x1876834907422080/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789891148 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 1974.117701] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 1974.121707] Lustre: Skipped 3 previous similar messages
[ 1978.207129] Lustre: 3622:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789891137/real 1789891137]  req@ffff906990ecd500 x1876834907422592/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789891153 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 1978.213891] Lustre: 3622:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 1978.248507] Lustre: server umount lustre-OST0000 complete
[ 1979.540907] Lustre: server umount lustre-OST0001 complete
[ 1984.557633] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local
[ 1988.185856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 1988.313612] LustreError: 115749:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 1988.322787] LustreError: 115749:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 26 previous similar messages
[ 1988.338796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 1989.656592] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1992.411657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 1992.527925] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 1993.922306] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1994.990063] Lustre: 116902:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 1997.357294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 1997.466197] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 1998.497515] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1667 to 0x280000401:1697)
[ 1999.412620] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2002.217705] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 2003.299457] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1668 to 0x2c0000401:1697)
[ 2003.301330] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:97)
[ 2004.167031] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2007.148745] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 2008.253612] Lustre: 118780:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 2018.271471] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 2018.274938] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2018.280658] Lustre: Skipped 1 previous similar message
[ 2018.282634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 2018.784087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 2018.786129] Lustre: Skipped 2 previous similar messages
[ 2023.763920] Lustre: server umount lustre-MDT0000 complete
[ 2024.976731] LustreError: 115732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789891199 with bad export cookie 6651697752048290097
[ 2024.978563] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2024.981257] LustreError: 115732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 2025.080396] Lustre: server umount lustre-MDT0001 complete
[ 2036.224052] Lustre: server umount lustre-OST0000 complete
[ 2047.752191] Lustre: server umount lustre-OST0001 complete
[ 2052.237073] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local
[ 2055.276137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2055.416630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2055.419400] Lustre: Skipped 1 previous similar message
[ 2056.633584] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2059.393715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2060.784155] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2061.768909] Lustre: 122660:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 2063.908935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 2065.904888] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2066.081890] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1667 to 0x280000401:1729)
[ 2068.689189] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 2069.794185] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1668 to 0x2c0000401:1729)
[ 2069.796492] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:129)
[ 2070.485296] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2073.285631] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 2074.291742] Lustre: 124526:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 2077.395796] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 04:00:51 (1789891251)
[ 2077.907941] Lustre: DEBUG MARKER: OST0_SIZE: 3605412 required: 6144
[ 2078.382546] Lustre: DEBUG MARKER: run for 4MB test file
[ 2083.226637] Lustre: DEBUG MARKER: User quota (limit: 4 MB)
[ 2085.082064] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT
[ 2085.600316] Lustre: DEBUG MARKER: Write half of file
[ 2086.152695] Lustre: DEBUG MARKER: Write out of block quota ...
[ 2086.662763] Lustre: DEBUG MARKER: Step1: done
[ 2087.162848] Lustre: DEBUG MARKER: Step2: rewrite should succeed
[ 2087.707440] Lustre: DEBUG MARKER: Step2: done
[ 2100.555462] Lustre: DEBUG MARKER: OST0_SIZE: 3605412 required: 61440
[ 2101.055135] Lustre: DEBUG MARKER: run for 40MB test file
[ 2104.604456] Lustre: DEBUG MARKER: User quota (limit: 40 MB)
[ 2106.435757] Lustre: DEBUG MARKER: Files for user (quota_usr), count=0:
[ 2107.024665] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2048).
[ 2107.554517] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT
[ 2108.051774] Lustre: DEBUG MARKER: Write half of file
[ 2108.945916] Lustre: DEBUG MARKER: Write out of block quota ...
[ 2109.810815] Lustre: DEBUG MARKER: Step1: done
[ 2110.307428] Lustre: DEBUG MARKER: Step2: rewrite should succeed
[ 2110.834754] Lustre: DEBUG MARKER: Step2: done
[ 2132.210202] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 04:01:46 (1789891306)
[ 2151.530096] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 04:02:05 (1789891325)
[ 2163.352821] Lustre: DEBUG MARKER: sanity-quota test_25: @@@@@@ FAIL: Used space(45056) for user quota_usr isn't 0.
[ 2176.645030] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 04:02:30 (1789891350)
[ 2178.911109] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 04:02:33 (1789891353)
[ 2183.216164] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 04:02:37 (1789891357)
[ 2186.678596] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 04:02:40 (1789891360)
[ 2189.781325] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 04:02:44 (1789891364)
[ 2217.808296] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 2221.016373] Lustre: DEBUG MARKER: sanity-quota test_33: @@@@@@ FAIL: Used space (15360) for user 60000 isn't 0.
[ 2226.802981] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 2229.872201] Lustre: DEBUG MARKER: sanity-quota test_34: @@@@@@ FAIL: Used space (15360) for user 60000 isn't 0.
[ 2234.920364] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 04:03:29 (1789891409)
[ 2249.302028] Lustre: DEBUG MARKER: sanity-quota test_35: @@@@@@ FAIL: Used inodes for user 60000 is 0, expected 1
[ 2263.721383] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 04:03:57 (1789891437)
[ 2267.121845] Lustre: DEBUG MARKER: sanity-quota test_37: @@@@@@ FAIL: Used space (15360) for user 60000 isn't 0.
[ 2272.278095] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 04:04:06 (1789891446)
[ 2275.644253] Lustre: DEBUG MARKER: sanity-quota test_38: @@@@@@ FAIL: Used space (15360) for user 60000 isn't 0.
[ 2281.052425] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 04:04:15 (1789891455)
[ 2285.024463] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 2285.025364] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2285.026144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 2285.031637] Lustre: Skipped 6 previous similar messages
[ 2290.145759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 2290.147849] Lustre: Skipped 6 previous similar messages
[ 2290.210471] Lustre: server umount lustre-MDT0000 complete
[ 2291.434092] LustreError: 121502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789891466 with bad export cookie 6651697752048292407
[ 2291.437447] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2291.440296] LustreError: 121502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 2291.554903] Lustre: server umount lustre-MDT0001 complete
[ 2302.463348] Lustre: server umount lustre-OST0000 complete
[ 2314.010916] Lustre: server umount lustre-OST0001 complete
[ 2319.326791] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local
[ 2322.991608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2323.130333] LustreError: 143434:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2323.137228] LustreError: 143434:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages
[ 2323.155485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2323.159130] Lustre: Skipped 3 previous similar messages
[ 2324.565811] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2327.471959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2328.951517] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2329.926780] Lustre: 144579:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 2332.070806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 2333.217382] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1734 to 0x280000401:1761)
[ 2334.020801] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2336.724399] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 2337.825941] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1732 to 0x2c0000401:1761)
[ 2337.827208] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:161)
[ 2338.695646] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2341.800659] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 2342.964654] Lustre: 146450:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 2357.603288] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 04:05:31 (1789891531)
[ 2372.334139] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 04:05:46 (1789891546)
[ 2388.833230] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 04:06:03 (1789891563)
[ 2392.508684] Lustre: DEBUG MARKER: sanity-quota test_40c: @@@@@@ FAIL: file count expected 3 got 2
[ 2408.318716] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 04:06:22 (1789891582)
[ 2424.410771] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 04:06:38 (1789891598)
[ 2459.964282] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 04:07:14 (1789891634)
[ 2470.461383] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 04:07:24 (1789891644)
[ 2495.815308] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 04:07:50 (1789891670)
[ 2497.526320] Lustre: *** cfs_fail_loc=a09, val=0***
[ 2498.028409] Lustre: *** cfs_fail_loc=a09, val=0***
[ 2498.029561] Lustre: Skipped 105 previous similar messages
[ 2499.031343] Lustre: *** cfs_fail_loc=a09, val=0***
[ 2499.032566] Lustre: Skipped 215 previous similar messages
[ 2501.036442] Lustre: *** cfs_fail_loc=a09, val=0***
[ 2501.037520] Lustre: Skipped 423 previous similar messages
[ 2505.048119] Lustre: *** cfs_fail_loc=a09, val=0***
[ 2505.049712] Lustre: Skipped 825 previous similar messages
[ 2513.048540] Lustre: *** cfs_fail_loc=a09, val=0***
[ 2513.049594] Lustre: Skipped 1690 previous similar messages
[ 2529.049138] Lustre: *** cfs_fail_loc=a09, val=0***
[ 2529.050639] Lustre: Skipped 3228 previous similar messages
[ 2650.080893] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 2650.082867] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2650.084084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 2650.087381] Lustre: Skipped 2 previous similar messages
[ 2650.089561] Lustre: Skipped 1 previous similar message
[ 2652.423298] Lustre: server umount lustre-MDT0000 complete
[ 2653.760415] LustreError: 143415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789891828 with bad export cookie 6651697752048299869
[ 2653.763393] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2653.764258] LustreError: 143415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages
[ 2655.199698] LustreError: 143463:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2655.205261] LustreError: 143463:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages
[ 2658.207895] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping)
[ 2658.209822] Lustre: Skipped 3 previous similar messages
[ 2659.953711] Lustre: server umount lustre-MDT0001 complete
[ 2661.486255] Lustre: server umount lustre-OST0000 complete
[ 2662.943759] Lustre: server umount lustre-OST0001 complete
[ 2665.273557] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid
[ 2667.983805] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local
[ 2685.156854] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local
[ 2688.571505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2688.653155] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000
[ 2688.664063] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[ 2688.701473] Lustre: lustre-MDT0000: new disk, initializing
[ 2688.720850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2688.724463] Lustre: Skipped 3 previous similar messages
[ 2688.730101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[ 2690.053630] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2694.152374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2694.186886] Lustre: 163880:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17
[ 2694.200657] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
[ 2694.202313] Lustre: Skipped 1 previous similar message
[ 2694.236670] Lustre: lustre-MDT0001: new disk, initializing
[ 2694.260295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt
[ 2694.263773] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt]
[ 2695.554357] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2697.807591] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params
[ 2699.938685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 2700.022050] Lustre: lustre-OST0000: new disk, initializing
[ 2700.024696] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[ 2700.027103] Lustre: 165516:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0]
[ 2701.101462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost
[ 2701.104121] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost]
[ 2701.123087] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401
[ 2702.076574] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2706.191673] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 2706.236134] Lustre: lustre-OST0001: new disk, initializing
[ 2706.238503] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[ 2706.241493] Lustre: 166387:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0]
[ 2707.435540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost
[ 2707.438267] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost]
[ 2707.455267] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401
[ 2708.168092] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2712.627492] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 2713.827686] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params
[ 2723.564598] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 04:11:37 (1789891897)
[ 2725.789983] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 04:11:40 (1789891900)
[ 2742.387413] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 04:11:56 (1789891916)
[ 2763.499627] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 04:12:17 (1789891937)
[ 2771.028595] Lustre: DEBUG MARKER: rename directory return 255
[ 2788.450430] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 04:12:42 (1789891962)
[ 2795.805956] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 04:12:50 (1789891970)
[ 2805.135763] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 04:12:59 (1789891979)
[ 2836.013795] LustreError: 168476:0:(qmt_entry.c:559:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-0x0 id:60001 enforced:1 hard:51200 soft:0 granted:51200 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[ 2850.749865] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 04:13:45 (1789892025)
[ 2860.189595] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 04:13:54 (1789892034)
[ 2875.852938] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 04:14:10 (1789892050)
[ 2926.514864] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 04:15:00 (1789892100)
[ 2927.586855] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2927.589951] Lustre: Skipped 3 previous similar messages
[ 2927.592459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 2930.143663] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 2933.791190] Lustre: server umount lustre-MDT0000 complete
[ 2935.001039] LustreError: 163873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789892109 with bad export cookie 6651697752048672066
[ 2935.003883] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2935.007219] LustreError: 163873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 2935.124149] Lustre: server umount lustre-MDT0001 complete
[ 2947.075770] Lustre: server umount lustre-OST0000 complete
[ 2958.532544] Lustre: server umount lustre-OST0001 complete
[ 2969.175218] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local
[ 2975.830986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2976.124244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2976.129595] Lustre: Skipped 3 previous similar messages
[ 2976.136151] LustreError: lustre-MDT0000: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[ 2979.049736] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2982.992563] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2983.111917] LustreError: lustre-MDT0001: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[ 2984.675142] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2985.697587] Lustre: 185184:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 2987.959810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 2988.053081] LustreError: lustre-OST0000: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[ 2990.096089] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2993.080848] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 2993.128422] LustreError: lustre-OST0001: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[ 2995.118254] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3000.290519] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:20 to 0x2c0000401:65)
[ 3000.290769] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:20 to 0x280000401:65)
[ 3003.412127] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 3007.239343] LustreError: 184036:0:(osd_handler.c:3426:osd_quota_transfer()) lustre-MDT0000: quota transfer failed. Is project enforcement enabled on the ldiskfs filesystem? rc = -95
[ 3008.991964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 3008.994299] Lustre: Skipped 6 previous similar messages
[ 3014.545906] Lustre: server umount lustre-MDT0000 complete
[ 3015.852445] LustreError: 184020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789892190 with bad export cookie 6651697752048720639
[ 3015.855364] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3015.856894] LustreError: 184020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 3015.961184] Lustre: server umount lustre-MDT0001 complete
[ 3027.970294] Lustre: server umount lustre-OST0000 complete
[ 3039.764171] Lustre: server umount lustre-OST0001 complete
[ 3046.138802] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local
[ 3049.921966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3051.429144] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3054.354684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3055.821023] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3056.850205] Lustre: 190590:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 3056.854476] Lustre: 190590:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message
[ 3059.147382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 3060.260646] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:20 to 0x280000401:97)
[ 3061.123247] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3064.240907] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 3065.317442] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97)
[ 3066.393396] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3069.459522] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 3084.765492] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 04:17:39 (1789892259)
[ 3106.024482] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[ 3106.555105] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 04:18:00 (1789892280)
[ 3113.331085] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[ 3113.815060] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 04:18:08 (1789892288)
[ 3118.017796] Lustre: DEBUG MARKER: sanity-quota test_64: @@@@@@ FAIL: expected 0 got 6
[ 3131.161901] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[ 3131.673565] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 04:18:25 (1789892305)
[ 3151.465672] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 04:18:45 (1789892325)
[ 3155.164521] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[ 3157.079993] Lustre: 199640:0:(qsd_reint.c:249:qsd_reint_index()) lustre-OST0000: index version for fid [0x200000005:0x1007:0x0] is 0, but index isn't empty (1)
[ 3157.084135] Lustre: 199640:0:(qsd_reint.c:249:qsd_reint_index()) Skipped 2 previous similar messages
[ 3158.501241] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 3160.261347] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 3161.388143] Lustre: DEBUG MARKER: Write...
[ 3171.227223] LustreError: 200983:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[ 3174.896372] Lustre: DEBUG MARKER: Write...
[ 3178.620204] Lustre: DEBUG MARKER: Write...
[ 3198.054936] Lustre: DEBUG MARKER: Files for user (quota_usr), count=0:
[ 3198.536034] Lustre: DEBUG MARKER: sanity-quota test_67: @@@@@@ FAIL: user quota isn't released after deletion
[ 3218.574337] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 04:19:52 (1789892392)
[ 3228.999083] LustreError: 205426:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[ 3249.116218] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 04:20:23 (1789892423)
[ 3258.669481] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[ 3259.148859] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[ 3289.636672] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 04:21:03 (1789892463)
[ 3309.351097] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 04:21:23 (1789892483)
[ 3327.879757] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 04:21:42 (1789892502)
[ 3331.423512] Lustre: DEBUG MARKER: User quota (block hardlimit:100 MB)
[ 3343.853620] Lustre: DEBUG MARKER: sanity-quota test_71a: @@@@@@ FAIL: Used space(10240) for user quota_usr isn't 0.
[ 3364.658298] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 04:22:18 (1789892538)
[ 3368.151615] Lustre: DEBUG MARKER: User quota (block hardlimit:1000 MB)
[ 3379.883789] Lustre: DEBUG MARKER: sanity-quota test_71b: @@@@@@ FAIL: Used space(10240) for user quota_usr isn't 0.
[ 3397.886829] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 04:22:52 (1789892572)
[ 3401.452233] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB)
[ 3408.680584] Lustre: DEBUG MARKER: sanity-quota test_72: @@@@@@ FAIL: Used space(10240) for user quota_usr isn't 0.