[ 1815.695938] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 1816.663559] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 1818.500732] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 1818.830037] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 1819.268221] Lustre: Failing over lustre-MDT0000
[ 1819.326741] Lustre: server umount lustre-MDT0000 complete
[ 1819.924094] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 1819.927383] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 1819.931203] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 1819.934335] LustreError: Skipped 2 previous similar messages
[ 1832.118040] LDISKFS-fs (dm-0): recovery complete
[ 1832.119681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 1832.152218] LustreError: 166-1: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 1832.152386] LustreError: 11-0: MGC192.168.201.105@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107
[ 1832.156722] Lustre: Evicted from MGS (at 192.168.201.105@tcp) after server handle changed from 0x98479445319bb47a to 0x9847944531c9284f
[ 1832.159558] Lustre: MGC192.168.201.105@tcp: Connection restored to 192.168.201.105@tcp (at 0@lo)
[ 1832.162848] Lustre: Skipped 1 previous similar message
[ 1832.215857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 1832.223177] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 1832.987043] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 1835.054986] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 1837.227390] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted.
[ 1837.243092] Lustre: lustre-OST0000: deleting orphan objects from 0x280000401:4015 to 0x280000401:4033
[ 1837.243099] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000401:4012 to 0x2c0000401:4033
[ 1837.731865] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1838.101934] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1840.169246] Lustre: DEBUG MARKER: (dd_pid=15571, time=0, timeout=600)
[ 1845.796443] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 1846.794066] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 1848.651967] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 1848.988063] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 1849.441973] Lustre: Failing over lustre-MDT0000
[ 1849.490709] Lustre: server umount lustre-MDT0000 complete
[ 1862.225857] LDISKFS-fs (dm-0): recovery complete
[ 1862.227266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 1862.259868] LustreError: 166-1: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 1862.260311] LustreError: 11-0: MGC192.168.201.105@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107
[ 1862.264921] Lustre: Evicted from MGS (at 192.168.201.105@tcp) after server handle changed from 0x9847944531c9284f to 0x9847944531c931b0
[ 1862.324796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 1863.002069] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 1867.354218] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000401:4012 to 0x2c0000401:4065
[ 1867.354483] Lustre: lustre-OST0000: deleting orphan objects from 0x280000401:4035 to 0x280000401:4065
[ 1867.857096] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1868.226783] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1870.232833] Lustre: DEBUG MARKER: (dd_pid=17852, time=0, timeout=600)
[ 1879.884587] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 08:35:27 (1694435727)
[ 1895.554303] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 08:35:43 (1694435743)
[ 1901.932320] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 1904.568010] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 1905.002812] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 1905.039182] LustreError: 10076:0:(qsd_reint.c:635:qqi_reint_delayed()) lustre-OST0001: Delaying reintegration for qtype:1 until pending updates are flushed.
[ 1905.043315] LustreError: 10076:0:(qsd_reint.c:635:qqi_reint_delayed()) Skipped 7 previous similar messages
[ 1905.427925] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 1905.875203] Lustre: DEBUG MARKER: Set quota for 1 times
[ 1907.254801] Lustre: DEBUG MARKER: Set quota for 2 times
[ 1908.660067] Lustre: DEBUG MARKER: Set quota for 3 times
[ 1910.008843] Lustre: DEBUG MARKER: Set quota for 4 times
[ 1911.429117] Lustre: DEBUG MARKER: Set quota for 5 times
[ 1912.807828] Lustre: DEBUG MARKER: Set quota for 6 times
[ 1914.190162] Lustre: DEBUG MARKER: Set quota for 7 times
[ 1915.562866] Lustre: DEBUG MARKER: Set quota for 8 times
[ 1916.920050] Lustre: DEBUG MARKER: Set quota for 9 times
[ 1918.289878] Lustre: DEBUG MARKER: Set quota for 10 times
[ 1919.635260] Lustre: DEBUG MARKER: Set quota for 11 times
[ 1921.008067] Lustre: DEBUG MARKER: Set quota for 12 times
[ 1922.385399] Lustre: DEBUG MARKER: Set quota for 13 times
[ 1923.764285] Lustre: DEBUG MARKER: Set quota for 14 times
[ 1925.124354] Lustre: DEBUG MARKER: Set quota for 15 times
[ 1926.501846] Lustre: DEBUG MARKER: Set quota for 16 times
[ 1927.860322] Lustre: DEBUG MARKER: Set quota for 17 times
[ 1929.195284] Lustre: DEBUG MARKER: Set quota for 18 times
[ 1930.551482] Lustre: DEBUG MARKER: Set quota for 19 times
[ 1931.906772] Lustre: DEBUG MARKER: Set quota for 20 times
[ 1933.251525] Lustre: DEBUG MARKER: Set quota for 21 times
[ 1934.657742] Lustre: DEBUG MARKER: Set quota for 22 times
[ 1951.434993] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 08:36:38 (1694435798)
[ 1957.476307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 1957.477732] Lustre: Skipped 3 previous similar messages
[ 1961.524426] Lustre: server umount lustre-MDT0000 complete
[ 1962.445247] LustreError: 6507:0:(ldlm_lockd.c:2572:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1694435809 with bad export cookie 10972902042020426160
[ 1962.446275] LustreError: 166-1: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 1962.452417] LustreError: 6507:0:(ldlm_lockd.c:2572:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 1962.484111] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 1962.487568] LustreError: Skipped 31 previous similar messages
[ 1962.488926] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping)
[ 1962.491044] Lustre: Skipped 1 previous similar message
[ 1962.545347] Lustre: server umount lustre-MDT0001 complete
[ 1969.517482] Lustre: server umount lustre-OST0000 complete
[ 1970.481709] Lustre: server umount lustre-OST0001 complete
[ 1973.333977] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local
[ 1975.375377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 1975.519014] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 1976.202681] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 1978.194235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 1978.993841] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 1980.993764] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5
[ 1980.997064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 1981.097790] mount.lustre (14834) used greatest stack depth: 10224 bytes left
[ 1982.176160] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 1984.112607] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5
[ 1984.115884] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 1984.168029] mount.lustre (15822) used greatest stack depth: 10016 bytes left
[ 1985.155639] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000401:4067 to 0x2c0000401:4097
[ 1985.157684] Lustre: lustre-OST0000: deleting orphan objects from 0x280000401:4068 to 0x280000401:4097
[ 1985.292458] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 1988.155592] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:3 to 0x280000400:97
[ 1991.452680] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 1992.233070] Lustre: Modifying parameter general.lod.*.mdt_hash in log params
[ 2008.180273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 2008.181933] Lustre: Skipped 3 previous similar messages
[ 2010.397489] Lustre: server umount lustre-MDT0000 complete
[ 2011.291161] LustreError: 13094:0:(ldlm_lockd.c:2572:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1694435858 with bad export cookie 10972902042020434308
[ 2011.291996] LustreError: 166-1: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2011.302218] LustreError: 13094:0:(ldlm_lockd.c:2572:ldlm_cancel_handler()) Skipped 3 previous similar messages
[ 2011.382729] Lustre: server umount lustre-MDT0001 complete
[ 2022.310705] Lustre: 18341:0:(client.c:2310:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1694435859/real 1694435859]  req@ffff8800a0f9af80 x1776742700261184/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1694435869 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0
[ 2022.327186] Lustre: server umount lustre-OST0000 complete
[ 2033.192742] Lustre: 18614:0:(client.c:2310:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1694435870/real 1694435870]  req@ffff88009d58b440 x1776742700261696/t0(0) o39->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1694435880 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0
[ 2033.246245] Lustre: server umount lustre-OST0001 complete
[ 2036.143701] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local
[ 2038.186568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2038.336244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2038.338928] Lustre: Skipped 3 previous similar messages
[ 2039.007205] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 2041.002592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2041.114006] mount.lustre (20865) used greatest stack depth: 9968 bytes left
[ 2041.768480] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 2043.797319] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5
[ 2043.800757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 2043.895035] mount.lustre (21719) used greatest stack depth: 9536 bytes left
[ 2044.976280] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 2047.201189] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5
[ 2047.209686] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 2048.300522] Lustre: lustre-OST0000: deleting orphan objects from 0x280000401:4068 to 0x280000401:4129
[ 2048.316883] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000401:4067 to 0x2c0000401:4129
[ 2048.803817] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 2051.299490] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:3 to 0x280000400:129
[ 2054.988853] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 2055.757302] Lustre: Modifying parameter general.lod.*.mdt_hash in log params
[ 2064.071994] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 08:38:31 (1694435911)
[ 2064.406025] Lustre: DEBUG MARKER: OST0_SIZE: 3605408 required: 6144
[ 2064.722296] Lustre: DEBUG MARKER: run for 4MB test file
[ 2069.496480] Lustre: DEBUG MARKER: User quota (limit: 4 MB)
[ 2070.590819] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT
[ 2070.975963] Lustre: DEBUG MARKER: Write half of file
[ 2071.417812] Lustre: DEBUG MARKER: Write out of block quota ...
[ 2071.794462] Lustre: DEBUG MARKER: Step1: done
[ 2072.154420] Lustre: DEBUG MARKER: Step2: rewrite should succeed
[ 2072.509380] Lustre: DEBUG MARKER: Step2: done
[ 2082.536920] Lustre: DEBUG MARKER: OST0_SIZE: 3605408 required: 61440
[ 2082.875426] Lustre: DEBUG MARKER: run for 40MB test file
[ 2085.439098] Lustre: DEBUG MARKER: User quota (limit: 40 MB)
[ 2086.679577] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT
[ 2087.074796] Lustre: DEBUG MARKER: Write half of file
[ 2087.841266] Lustre: DEBUG MARKER: Write out of block quota ...
[ 2088.595886] Lustre: DEBUG MARKER: Step1: done
[ 2088.911194] Lustre: DEBUG MARKER: Step2: rewrite should succeed
[ 2089.281487] Lustre: DEBUG MARKER: Step2: done
[ 2105.558114] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 08:39:13 (1694435953)
[ 2119.934180] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 08:39:27 (1694435967)
[ 2129.488246] Lustre: DEBUG MARKER: Write...
[ 2129.926281] Lustre: DEBUG MARKER: Write out of block quota ...
[ 2137.118875] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 2137.537902] Lustre: DEBUG MARKER: sanity-quota test_25: @@@@@@ FAIL: user write success, but expect EDQUOT