[ 5718.263980] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5725.435709] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 5729.529101] LustreError: 119511:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5730.646526] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5732.817485] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5736.187546] Lustre: Failing over lustre-MDT0000
[ 5736.809265] Lustre: server umount lustre-MDT0000 complete
[ 5754.271221] Lustre: 3329:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780336291/real 1780336291]  req@ffff9e688798ce00 x1866812087314944/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1780336307 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 5754.306975] Lustre: 3329:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5754.316844] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5756.385785] Lustre: 3329:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780336293/real 1780336293]  req@ffff9e68cb6f8000 x1866812087315072/t0(0) o400->MGC192.168.204.148@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780336309 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5756.442868] Lustre: 3329:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5756.459943] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5761.631170] Lustre: 3328:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780336298/real 1780336298]  req@ffff9e68cb6fb800 x1866812087315712/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780336314 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5761.660840] Lustre: 3328:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5767.312492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5767.411382] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5770.719781] Lustre: 3329:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780336307/real 1780336307]  req@ffff9e69ba5d2680 x1866812087316608/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780336323 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5770.775445] Lustre: 3329:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5773.714876] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5777.929478] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5778.108481] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5778.173378] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:578 to 0x240000400:609)
[ 5778.186387] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:574 to 0x280000400:609)
[ 5779.558281] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[ 5784.145434] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5786.899567] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5800.472132] Lustre: DEBUG MARKER: (dd_pid=113901, time=6, timeout=600)
[ 5855.322092] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5861.998648] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 5865.200828] LustreError: 122270:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5866.239497] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5867.885628] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5870.007401] Lustre: Failing over lustre-MDT0000
[ 5870.156076] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.48@tcp (stopping)
[ 5870.341418] Lustre: server umount lustre-MDT0000 complete
[ 5886.431156] Lustre: 3329:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780336423/real 1780336423]  req@ffff9e688685e680 x1866812087359872/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1780336439 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 5886.479720] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5886.494772] Lustre: Skipped 1 previous similar message
[ 5888.021791] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5888.707403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5888.797852] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5889.550349] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5889.739439] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5889.786223] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:611 to 0x240000400:641)
[ 5889.786248] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:574 to 0x280000400:641)
[ 5891.572707] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 5891.581643] Lustre: Skipped 1 previous similar message
[ 5893.455828] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5901.359533] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5903.258661] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5916.715830] Lustre: DEBUG MARKER: (dd_pid=116317, time=7, timeout=600)
[ 5984.305768] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 13:55:35 (1780336535)
[ 6008.552119] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 6010.456331] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 6017.309841] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 6019.327990] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 6055.662448] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 13:56:46 (1780336606)
[ 6086.705696] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 6105.853924] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 6108.055976] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 6109.843556] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 6112.125271] Lustre: DEBUG MARKER: Set quota for 1 times
[ 6115.885691] Lustre: DEBUG MARKER: Set quota for 2 times
[ 6119.907755] Lustre: DEBUG MARKER: Set quota for 3 times
[ 6124.191686] Lustre: DEBUG MARKER: Set quota for 4 times
[ 6129.046819] Lustre: DEBUG MARKER: Set quota for 5 times
[ 6133.572876] Lustre: DEBUG MARKER: Set quota for 6 times
[ 6138.497896] Lustre: DEBUG MARKER: Set quota for 7 times
[ 6199.128861] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 13:59:10 (1780336750)
[ 6214.794147] Lustre: server umount lustre-MDT0000 complete
[ 6219.199393] LustreError: 5782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780336772 with bad export cookie 9209035176485965738
[ 6219.223568] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6229.555503] Lustre: server umount lustre-OST0000 complete
[ 6233.568132] Lustre: 3327:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780336770/real 1780336770]  req@ffff9e6982d30a80 x1866812087487616/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780336786 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6233.617324] Lustre: 3327:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
[ 6233.625172] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6233.635697] Lustre: Skipped 1 previous similar message
[ 6234.507749] Lustre: server umount lustre-OST0001 complete
[ 6253.252154] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local
[ 6263.983407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6269.582996] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6273.197357] Lustre: 131630:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6279.611367] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 6287.617292] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6288.804328] LustreError: 131947:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6288.833960] LustreError: 131947:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message
[ 6288.889045] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:643 to 0x240000400:673)
[ 6293.996213] LustreError: 131947:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6296.692252] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6301.706279] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:644 to 0x280000400:673)
[ 6305.542529] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6314.763395] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6318.762872] Lustre: 133428:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6336.049162] Lustre: server umount lustre-MDT0000 complete
[ 6340.415287] LustreError: 131193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780336893 with bad export cookie 9209035176485972633
[ 6340.439495] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6350.983223] Lustre: server umount lustre-OST0000 complete
[ 6353.887442] Lustre: 3329:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780336890/real 1780336890]  req@ffff9e69886eb800 x1866812087530624/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780336906 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6353.929192] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6355.396680] Lustre: server umount lustre-OST0001 complete
[ 6371.865681] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local
[ 6384.987850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6390.815642] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6394.784329] Lustre: 135993:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6405.696322] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6409.512363] LustreError: 136318:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6409.538409] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:643 to 0x240000400:705)
[ 6413.139865] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6413.147303] Lustre: Skipped 1 previous similar message
[ 6418.416262] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:644 to 0x280000400:705)
[ 6419.735674] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6427.211754] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6430.686049] Lustre: 137778:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6441.080938] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 14:03:11 (1780336991)
[ 6443.006435] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 6444.867675] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 14:03:15 (1780336995)
[ 6500.749659] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 14:04:11 (1780337051)
[ 6557.529392] Lustre: DEBUG MARKER: Write...
[ 6560.974827] Lustre: DEBUG MARKER: Write out of block quota ...
[ 6626.440687] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 14:06:17 (1780337177)
[ 6634.055289] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 14:06:24 (1780337184)
[ 6644.382132] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 14:06:35 (1780337195)
[ 6652.781753] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 14:06:43 (1780337203)
[ 6660.558688] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 14:06:51 (1780337211)
[ 6728.775718] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 6900.517522] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 7102.040564] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 14:14:12 (1780337652)
[ 7170.717063] Lustre: DEBUG MARKER: Restart...
[ 7176.173399] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 7176.176035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 7176.197279] Lustre: Skipped 1 previous similar message
[ 7181.167915] Lustre: server umount lustre-MDT0000 complete
[ 7186.207181] LustreError: 138530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780337739 with bad export cookie 9209035176485974159
[ 7186.235396] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 7186.243270] LustreError: 138530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 7186.348993] Lustre: server umount lustre-OST0000 complete
[ 7191.151581] Lustre: server umount lustre-OST0001 complete
[ 7206.560444] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local
[ 7216.078797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 7220.748875] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7224.213651] Lustre: 156444:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 7230.531904] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 7238.098123] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7240.737658] LustreError: 156762:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 7240.789372] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:715 to 0x240000400:737)
[ 7245.793627] LustreError: 156764:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 7247.505949] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 7252.989340] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:713 to 0x280000400:737)
[ 7254.410513] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7262.703513] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 7266.585565] Lustre: 158225:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 7335.861717] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 14:18:06 (1780337886)
[ 7412.025700] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 14:19:23 (1780337963)
[ 9458.701799] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 14:53:29 (1780340009)
[ 9476.064906] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 9476.073234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 9476.077953] Lustre: Skipped 1 previous similar message
[ 9476.104337] Lustre: Skipped 2 previous similar messages
[ 9478.216610] Lustre: server umount lustre-MDT0000 complete
[ 9483.251035] LustreError: 156012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780340036 with bad export cookie 9209035176485980662
[ 9483.253842] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 9483.279279] LustreError: 156012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 9483.402768] Lustre: server umount lustre-OST0000 complete
[ 9488.479334] Lustre: server umount lustre-OST0001 complete
[ 9506.087992] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local
[ 9516.730588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 9522.832387] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9527.041241] Lustre: 166708:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 9533.986799] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 9541.112682] LustreError: 167032:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9541.133304] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5740 to 0x240000400:5761)
[ 9541.472891] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9546.213646] LustreError: 167032:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9551.134047] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 9556.466594] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5738 to 0x280000400:5761)
[ 9557.678631] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9565.737248] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 9569.742543] Lustre: 168504:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 9611.268624] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 14:56:02 (1780340162)
[ 9665.345502] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 14:56:54 (1780340214)
[ 9714.102293] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 14:57:44 (1780340264)
[ 9715.935891] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs
[ 9718.065985] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 14:57:48 (1780340268)
[ 9719.988501] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs
[ 9722.422249] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 14:57:52 (1780340272)
[ 9831.790394] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 14:59:42 (1780340382)
[ 9890.459286] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 15:00:40 (1780340440)
[ 9992.056861] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 15:02:22 (1780340542)
[10005.279200] Lustre: *** cfs_fail_loc=a09, val=0***
[10005.289383] Lustre: Skipped 1 previous similar message
[10007.323475] Lustre: *** cfs_fail_loc=a09, val=0***
[10007.329305] Lustre: Skipped 65 previous similar messages
[10011.324144] Lustre: *** cfs_fail_loc=a09, val=0***
[10011.330989] Lustre: Skipped 176 previous similar messages
[10019.395292] Lustre: *** cfs_fail_loc=a09, val=0***
[10019.413982] Lustre: Skipped 260 previous similar messages
[10035.435844] Lustre: *** cfs_fail_loc=a09, val=0***
[10035.440044] Lustre: Skipped 551 previous similar messages
[10067.479172] Lustre: *** cfs_fail_loc=a09, val=0***
[10067.487784] Lustre: Skipped 1241 previous similar messages
[10131.500613] Lustre: *** cfs_fail_loc=a09, val=0***
[10131.503787] Lustre: Skipped 2425 previous similar messages
[10533.989225] Lustre: *** cfs_fail_loc=a08, val=0***
[10533.999934] Lustre: *** cfs_fail_loc=a08, val=0***
[10534.029317] Lustre: Skipped 3290 previous similar messages
[10839.149679] Lustre: *** cfs_fail_loc=a08, val=0***
[10839.153841] Lustre: *** cfs_fail_loc=a08, val=0***
[10839.160660] Lustre: Skipped 14 previous similar messages
[10839.188085] Lustre: Skipped 19 previous similar messages
[10844.266555] Lustre: *** cfs_fail_loc=a08, val=0***
[10844.274477] Lustre: Skipped 38 previous similar messages
[10849.393326] Lustre: *** cfs_fail_loc=a08, val=0***
[10849.405977] Lustre: Skipped 153 previous similar messages
[10854.518868] Lustre: *** cfs_fail_loc=a08, val=0***
[10854.526098] Lustre: Skipped 283 previous similar messages
[10864.743855] Lustre: *** cfs_fail_loc=a08, val=0***
[10864.759598] Lustre: Skipped 463 previous similar messages
[10884.244369] Lustre: *** cfs_fail_loc=a08, val=0***
[10884.248216] Lustre: Skipped 1056 previous similar messages
[10919.039689] Lustre: *** cfs_fail_loc=a08, val=0***
[10919.048930] Lustre: Skipped 1776 previous similar messages
[10969.570245] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[10969.571932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[10969.594146] Lustre: Skipped 1 previous similar message
[10969.608215] Lustre: Skipped 1 previous similar message
[10971.612256] Lustre: server umount lustre-MDT0000 complete
[10976.370399] LustreError: 166278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780341529 with bad export cookie 9209035176487763667
[10976.401100] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[10976.673398] Lustre: server umount lustre-OST0000 complete
[10982.224155] Lustre: server umount lustre-OST0001 complete
[10990.231809] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid
[10999.948332] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local
[11010.806992]  vdc: vdc1 vdc9
[11021.351797]  vde: vde1 vde9
[11031.945121]  vdf: vdf1 vdf9
[11049.847780] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local
[11059.770656] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000
[11060.064112] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[11060.225793] Lustre: lustre-MDT0000: new disk, initializing
[11060.651202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[11060.743069] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[11064.805282] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[11069.636546] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params
[11074.978244] Lustre: lustre-OST0000: new disk, initializing
[11074.981335] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[11074.994706] Lustre: Skipped 1 previous similar message
[11075.111293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[11076.819548] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost
[11076.829560] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost]
[11076.981130] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400
[11081.567581] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[11091.561862] Lustre: lustre-OST0001: new disk, initializing
[11091.565502] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[11091.669401] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[11093.585346] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost
[11093.592591] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost]
[11093.703596] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400
[11097.271484] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[11106.892855] Lustre: DEBUG MARKER: Using TIMEOUT=20
[11110.851510] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params
[11148.031604] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 15:21:39 (1780341699)
[11154.151974] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 15:21:45 (1780341705)
[11200.719656] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 15:22:32 (1780341752)
[11256.587790] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 15:23:27 (1780341807)
[11281.612438] Lustre: DEBUG MARKER: rename directory return 255
[11322.090801] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 15:24:33 (1780341873)
[11348.786304] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 15:25:00 (1780341900)
[11383.743259] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 15:25:35 (1780341935)
[11451.079307] LustreError: 183328:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-0x0 id:60001 enforced:1 hard:51200 soft:0 granted:51200 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[11483.981398] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 15:27:15 (1780342035)
[11513.839040] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 15:27:45 (1780342065)
[11552.556531] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 15:28:23 (1780342103)
[11690.368498] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 15:30:41 (1780342241)
[11691.432384] Lustre: DEBUG MARKER: SKIP: sanity-quota test_59 ldiskfs only test
[11692.738666] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 15:30:44 (1780342244)
[11739.556202] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[11740.701779] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 15:31:32 (1780342292)
[11765.851570] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[11766.989341] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 15:31:58 (1780342318)
[11807.182874] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[11808.203950] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 15:32:39 (1780342359)
[11845.009235] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 15:33:16 (1780342396)
[11846.365986] Lustre: DEBUG MARKER: SKIP: sanity-quota test_67 ZFS grants some block space together with inode
[11847.767342] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 15:33:19 (1780342399)
[11870.706278] LustreError: 206543:0:(mgs_handler.c:1140:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[11904.447676] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 15:34:15 (1780342455)
[11925.072621] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[11926.318631] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[12067.657777] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 15:36:59 (1780342619)
[12108.142323] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 15:37:39 (1780342659)
[12136.135582] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 15:38:07 (1780342687)
[12137.191785] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71a ZFS grants some block space together with inode
[12138.538583] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 15:38:09 (1780342689)
[12139.746652] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71b ZFS grants some block space together with inode
[12141.016870] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 15:38:12 (1780342692)
[12154.032419] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB)
[12165.537320] Lustre: DEBUG MARKER: Write...
[12167.645889] Lustre: DEBUG MARKER: Write out of block quota ...
[12217.807115] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 15:39:29 (1780342769)
[12239.636822] Lustre: DEBUG MARKER: set to use default quota
[12241.100192] Lustre: DEBUG MARKER: set default quota
[12242.260433] Lustre: DEBUG MARKER: get default quota
[12246.618822] Lustre: DEBUG MARKER: Test not out of quota
[12249.632484] Lustre: DEBUG MARKER: Test out of quota
[12258.419652] Lustre: DEBUG MARKER: Increase default quota
[12290.219416] Lustre: DEBUG MARKER: Set quota to override default quota
[12290.346046] LustreError: 208793:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45063 time:1780947643 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[12314.732271] Lustre: DEBUG MARKER: Set to use default quota again
[12347.296962] Lustre: DEBUG MARKER: Cleanup
[12425.938461] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 15:42:56 (1780342976)
[12453.685841] Lustre: DEBUG MARKER: set default quota for qpool1
[12455.591862] Lustre: DEBUG MARKER: Write from user that hasn't lqe
[12512.052861] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 15:44:23 (1780343063)
[12598.551493] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 15:45:49 (1780343149)
[12679.267578] Lustre: DEBUG MARKER: Write...
[12683.642743] Lustre: DEBUG MARKER: Write out of block quota ...
[12815.612468] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 15:49:26 (1780343366)
[12862.364548] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 15:50:13 (1780343413)
[12869.800930] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 15:50:20 (1780343420)
[12871.433303] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78A need >= 2.13.57 and ldiskfs for fallocate
[12873.378545] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 15:50:24 (1780343424)
[12874.935385] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78a need >= 2.13.57 and ldiskfs for fallocate
[12876.686821] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 15:50:27 (1780343427)
[12897.983579] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 15:50:48 (1780343448)
[12899.488477] Lustre: DEBUG MARKER: SKIP: sanity-quota test_80 ZFS grants some block space together with inode
[12901.392518] Lustre: DEBUG MARKER: == sanity-quota test 81: Race qmt_start_pool_recalc with qmt_pool_free ========================================================== 15:50:52 (1780343452)
[12916.997655] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[12923.817597] LustreError: 231959:0:(qmt_pool.c:1399:qmt_pool_recalc()) cfs_fail_timeout id a07 sleeping for 10000ms
[12926.433952] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[12926.434705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[12926.454097] Lustre: Skipped 1 previous similar message
[12926.470912] Lustre: Skipped 1 previous similar message
[12927.967807] LustreError: 3328:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED  req@ffff9e69ba4cce00 x1866812097609728/t0(0) o601->lustre-MDT0000-lwp-MDT0000@0@lo:23/10 lens 336/336 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[12928.001291] LustreError: 3328:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8  qsd:lustre-MDT0000 qtype:prj lqe: ffff9e68c63c3c80 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 375 qunit:0 qtune:0 edquot:0 default:no revoke:0
[12928.030430] LustreError: 3328:0:(qsd_handler.c:298:qsd_req_completion()) Skipped 3 previous similar messages
[12929.564517] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.48@tcp (stopping)
[12931.553265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[12931.565578] Lustre: Skipped 1 previous similar message
[12933.879137] LustreError: 231959:0:(qmt_pool.c:1399:qmt_pool_recalc()) cfs_fail_timeout id a07 awake
[12934.212386] Lustre: server umount lustre-MDT0000 complete
[12941.031530] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[12941.539175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[12941.733467] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:27 to 0x240000400:65)
[12941.739186] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:27 to 0x280000400:65)
[12946.087607] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[12956.132151] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[12956.139331] Lustre: Skipped 1 previous similar message
[12978.715885] Lustre: DEBUG MARKER: == sanity-quota test 82: verify more than 8 qids for single operation ========================================================== 15:52:09 (1780343529)
[13009.647475] Lustre: DEBUG MARKER: == sanity-quota test 83: Setting default quota shouldn't affect grace time ========================================================== 15:52:40 (1780343560)
[13040.131971] Lustre: DEBUG MARKER: == sanity-quota test 84: Reset quota should fix the insane granted quota ========================================================== 15:53:11 (1780343591)
[13087.624399] Lustre: *** cfs_fail_loc=a08, val=0***
[13087.628133] Lustre: Skipped 3919 previous similar messages
[13087.635860] Lustre: *** cfs_fail_loc=a08, val=0***
[13087.642514] Lustre: Skipped 141 previous similar messages
[13191.055695] Lustre: DEBUG MARKER: == sanity-quota test 85: do not hung at write with the least_qunit ========================================================== 15:55:42 (1780343742)
[13267.561206] Lustre: DEBUG MARKER: == sanity-quota test 86: Pre-acquired quota should be released if quota is over limit ========================================================== 15:56:58 (1780343818)
[13339.977508] LustreError: 232487:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1780948693 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[13340.033354] LustreError: 3327:0:(qsd_reint.c:627:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:0 until pending updates are flushed.
[13347.626866] Lustre: DEBUG MARKER: sanity-quota test_86: @@@@@@ FAIL: succeed to create files, expect failed