[ 1928.361292] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 1930.681933] Lustre: DEBUG MARKER: Files for user (quota_usr), count=0:
[ 1931.307403] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:54629).
[ 1931.913922] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 1933.060489] LustreError: 85992:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 1933.412820] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 1934.098225] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 1934.763970] Lustre: Failing over lustre-MDT0000
[ 1935.188889] Lustre: server umount lustre-MDT0000 complete
[ 1947.513472] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 1947.617382] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 1947.624359] Lustre: Skipped 1 previous similar message
[ 1947.663697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 1947.691059] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 1948.335603] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 1948.347931] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 1948.363995] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2430 to 0x280000400:2465)
[ 1948.364040] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2446 to 0x240000400:2465)
[ 1948.903207] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1952.531930] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 1952.737953] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 1953.022053] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1954.784106] Lustre: 3278:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789891111/real 1789891111]  req@ffff9a3185c14a80 x1876834893087104/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789891127 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 1955.314295] Lustre: DEBUG MARKER: (dd_pid=88215, time=0, timeout=600)
[ 1959.904170] Lustre: 3279:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789891116/real 1789891116]  req@ffff9a3185c15880 x1876834893087360/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789891132 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 1959.913631] Lustre: 3279:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 1985.308166] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 1987.067739] Lustre: DEBUG MARKER: Files for user (quota_usr), count=0:
[ 1987.545270] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:102414).
[ 1988.029170] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 1988.969885] LustreError: 89068:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 1989.256444] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 1989.725580] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 1990.356099] Lustre: Failing over lustre-MDT0000
[ 1990.447126] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.12@tcp (stopping)
[ 1990.545385] Lustre: server umount lustre-MDT0000 complete
[ 2002.780132] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2002.903954] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2002.911965] Lustre: Skipped 1 previous similar message
[ 2002.953689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2002.979421] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 2004.320988] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2008.032199] Lustre: 3280:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789891164/real 1789891164]  req@ffff9a32b5e06a00 x1876834893115008/t0(0) o601->lustre-MDT0000-lwp-OST0001@0@lo:23/10 lens 336/336 e 0 to 1 dl 1789891180 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 2008.035060] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 2008.047165] Lustre: 3280:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 2008.050034] Lustre: Skipped 1 previous similar message
[ 2008.746732] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 2008.759378] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 2008.775171] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2467 to 0x240000400:2497)
[ 2008.775203] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2430 to 0x280000400:2497)
[ 2010.559335] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2011.070676] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2013.461360] Lustre: DEBUG MARKER: (dd_pid=91050, time=0, timeout=600)
[ 2014.176138] Lustre: 3277:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789891171/real 1789891171]  req@ffff9a31935dea00 x1876834893115776/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789891187 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2014.190613] Lustre: 3277:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 2053.617905] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 04:00:26 (1789891226)
[ 2065.400873] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 2065.896288] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:102414).
[ 2067.649677] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 2068.115279] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:102414).
[ 2088.809392] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 04:01:01 (1789891261)
[ 2089.883702] LustreError: 3279:0:(qsd_reint.c:633:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:0 until pending updates are flushed.
[ 2097.889124] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 2106.518546] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 2107.039352] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 2107.550309] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 2108.179409] Lustre: DEBUG MARKER: Set quota for 1 times
[ 2109.964461] Lustre: DEBUG MARKER: Set quota for 2 times
[ 2111.802086] Lustre: DEBUG MARKER: Set quota for 3 times
[ 2113.501308] Lustre: DEBUG MARKER: Set quota for 4 times
[ 2115.346927] Lustre: DEBUG MARKER: Set quota for 5 times
[ 2117.198792] Lustre: DEBUG MARKER: Set quota for 6 times
[ 2119.003037] Lustre: DEBUG MARKER: Set quota for 7 times
[ 2120.903863] Lustre: DEBUG MARKER: Set quota for 8 times
[ 2122.659684] Lustre: DEBUG MARKER: Set quota for 9 times
[ 2124.379949] Lustre: DEBUG MARKER: Set quota for 10 times
[ 2126.147402] Lustre: DEBUG MARKER: Set quota for 11 times
[ 2127.918104] Lustre: DEBUG MARKER: Set quota for 12 times
[ 2129.757962] Lustre: DEBUG MARKER: Set quota for 13 times
[ 2131.609643] Lustre: DEBUG MARKER: Set quota for 14 times
[ 2133.461305] Lustre: DEBUG MARKER: Set quota for 15 times
[ 2135.302991] Lustre: DEBUG MARKER: Set quota for 16 times
[ 2137.164266] Lustre: DEBUG MARKER: Set quota for 17 times
[ 2160.784775] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 04:02:13 (1789891333)
[ 2166.752662] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2166.752912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 2166.756086] Lustre: Skipped 1 previous similar message
[ 2166.760102] Lustre: Skipped 1 previous similar message
[ 2167.996429] Lustre: server umount lustre-MDT0000 complete
[ 2169.368980] LustreError: 5748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789891342 with bad export cookie 14388611505878732397
[ 2169.370364] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2169.373194] LustreError: 5748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 2169.419255] Lustre: server umount lustre-OST0000 complete
[ 2170.788787] Lustre: server umount lustre-OST0001 complete
[ 2175.446453] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local
[ 2178.953299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2180.362864] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2181.353805] Lustre: 99464:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 2183.086201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 2185.026178] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2187.290570] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 2188.329333] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2499 to 0x240000400:2529)
[ 2188.331296] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2500 to 0x280000400:2529)
[ 2189.250757] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2192.237324] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 2193.332848] Lustre: 101273:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 2198.496668] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2198.497137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 2198.501322] Lustre: Skipped 1 previous similar message
[ 2203.616476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 2203.619082] Lustre: Skipped 2 previous similar messages
[ 2204.137231] Lustre: server umount lustre-MDT0000 complete
[ 2205.436665] LustreError: 101523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789891378 with bad export cookie 14388611505878752704
[ 2205.437895] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2205.440712] LustreError: 101523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 2205.470709] Lustre: server umount lustre-OST0000 complete
[ 2206.835564] Lustre: server umount lustre-OST0001 complete
[ 2211.459721] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local
[ 2214.282765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2215.556950] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2216.501315] Lustre: 103871:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 2220.131883] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2224.493767] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2224.616738] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2499 to 0x240000400:2561)
[ 2224.617890] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2500 to 0x280000400:2561)
[ 2227.511468] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 2228.648139] Lustre: 105669:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 2231.943852] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 04:03:24 (1789891404)
[ 2232.428692] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 2232.963681] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 04:03:25 (1789891405)
[ 2265.020708] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 04:03:57 (1789891437)
[ 2283.337967] Lustre: DEBUG MARKER: sanity-quota test_25: @@@@@@ FAIL: Used space(6150) for user quota_usr isn't 0.