[ 5959.296380] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5965.758980] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 5969.458145] LustreError: 125227:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5970.800599] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5972.938054] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5974.980158] Lustre: Failing over lustre-MDT0000
[ 5975.233430] Lustre: server umount lustre-MDT0000 complete
[ 5993.442432] Lustre: 3312:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788468936/real 1788468936]  req@ffff8ea08694e680 x1875339462996736/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788468952 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5993.446269] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5993.481489] Lustre: 3312:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
[ 5993.498700] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5998.623200] Lustre: 3313:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788468941/real 1788468941]  req@ffff8ea087392300 x1875339462996992/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788468957 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5998.650851] Lustre: 3313:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 6002.783400] Lustre: 3313:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788468946/real 1788468946]  req@ffff8ea0cb060380 x1875339462997504/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788468962 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6003.682331] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xf8e8a4f93a8c3296
[ 6003.700497] Lustre: MGC192.168.202.128@tcp: Connection restored to 0@lo (at 0@lo)
[ 6004.289983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6004.369578] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 6005.354178] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 6005.507917] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 6005.572421] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:661 to 0x280000400:705)
[ 6005.574115] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:664 to 0x240000400:705)
[ 6008.336131] Lustre: 3311:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788468951/real 1788468951]  req@ffff8ea0c99ac380 x1875339462997888/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788468967 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6008.368483] Lustre: 3311:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 6009.250220] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6020.215720] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6022.210649] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6028.624463] Lustre: DEBUG MARKER: (dd_pid=117333, time=0, timeout=600)
[ 6082.761677] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 6090.910462] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 6093.982367] LustreError: 128148:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 6094.986604] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 6096.485130] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 6098.903812] Lustre: Failing over lustre-MDT0000
[ 6099.501456] Lustre: server umount lustre-MDT0000 complete
[ 6115.808607] Lustre: 3314:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788469059/real 1788469059]  req@ffff8ea0876ef100 x1875339463036416/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1788469075 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 6115.860843] Lustre: 3314:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 6115.876345] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6115.894585] Lustre: Skipped 1 previous similar message
[ 6116.832338] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6127.737039] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6127.778684] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 6129.236785] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 6129.487837] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 6129.549201] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:707 to 0x240000400:737)
[ 6129.551959] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:661 to 0x280000400:737)
[ 6131.937034] Lustre: 3311:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788469075/real 1788469075]  req@ffff8ea0876ef480 x1875339463037696/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788469091 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6131.955790] Lustre: 3311:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
[ 6134.132854] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6141.229278] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[ 6141.244585] Lustre: Skipped 2 previous similar messages
[ 6144.641425] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6146.605691] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6159.721386] Lustre: DEBUG MARKER: (dd_pid=119774, time=6, timeout=600)
[ 6237.603433] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 16:59:54 (1788469194)
[ 6264.220104] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 6266.393969] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 6272.545356] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 6274.542062] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 6309.505271] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 17:01:07 (1788469267)
[ 6341.766481] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 6360.096260] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 6362.659039] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 6364.736554] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 6367.462503] Lustre: DEBUG MARKER: Set quota for 1 times
[ 6372.721382] Lustre: DEBUG MARKER: Set quota for 2 times
[ 6377.620343] Lustre: DEBUG MARKER: Set quota for 3 times
[ 6382.683904] Lustre: DEBUG MARKER: Set quota for 4 times
[ 6386.621433] Lustre: DEBUG MARKER: Set quota for 5 times
[ 6391.440332] Lustre: DEBUG MARKER: Set quota for 6 times
[ 6440.162479] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 17:03:17 (1788469397)
[ 6454.752090] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6454.756033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6454.760243] Lustre: Skipped 2 previous similar messages
[ 6454.767374] Lustre: Skipped 1 previous similar message
[ 6455.533340] Lustre: server umount lustre-MDT0000 complete
[ 6461.014493] LustreError: 40022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788469420 with bad export cookie 17935766906339081186
[ 6461.025874] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6461.032926] LustreError: 40022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 6461.235629] Lustre: server umount lustre-OST0000 complete
[ 6465.611518] Lustre: server umount lustre-OST0001 complete
[ 6481.900903] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local
[ 6492.755774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6499.658918] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6504.449807] Lustre: 137705:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6511.856223] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 6516.963140] LustreError: 138077:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6517.025114] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:740 to 0x240000400:769)
[ 6518.821659] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6522.337656] LustreError: 138077:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6529.106269] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6534.148168] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:739 to 0x280000400:769)
[ 6536.946567] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6548.007954] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6555.720974] Lustre: 139639:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6575.072965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6575.082869] Lustre: Skipped 1 previous similar message
[ 6577.328610] Lustre: server umount lustre-MDT0000 complete
[ 6581.959246] LustreError: 137251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788469541 with bad export cookie 17935766906339089418
[ 6581.965476] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6581.977064] LustreError: 137251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 6582.143884] Lustre: server umount lustre-OST0000 complete
[ 6586.682084] Lustre: server umount lustre-OST0001 complete
[ 6603.155116] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local
[ 6614.771539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6620.308155] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6624.128245] Lustre: 142292:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6631.157123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 6638.077090] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6639.339543] LustreError: 142662:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6639.374314] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:740 to 0x240000400:801)
[ 6639.378263] LustreError: 142662:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message
[ 6651.933635] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:739 to 0x280000400:801)
[ 6653.976841] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6662.948145] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6666.769607] Lustre: 144203:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6681.652143] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 17:07:18 (1788469638)
[ 6684.502584] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 6687.225495] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 17:07:24 (1788469644)
[ 6747.190940] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 17:08:24 (1788469704)
[ 6798.394554] Lustre: DEBUG MARKER: Write...
[ 6801.722875] Lustre: DEBUG MARKER: Write out of block quota ...
[ 6869.965894] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 17:10:28 (1788469828)
[ 6878.888739] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 17:10:36 (1788469836)
[ 6890.938912] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 17:10:49 (1788469849)
[ 6903.422831] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 17:11:00 (1788469860)
[ 6914.792103] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 17:11:12 (1788469872)
[ 6987.732815] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 7162.608433] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 7373.671888] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 17:18:51 (1788470331)
[ 7443.062781] Lustre: DEBUG MARKER: Restart...
[ 7450.606390] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 7450.629701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 7450.665161] Lustre: Skipped 3 previous similar messages
[ 7450.696815] Lustre: Skipped 1 previous similar message
[ 7451.250312] Lustre: server umount lustre-MDT0000 complete
[ 7456.226039] LustreError: 144950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788470415 with bad export cookie 17935766906339090951
[ 7456.228404] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 7456.241329] LustreError: 144950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 7456.435495] Lustre: server umount lustre-OST0000 complete
[ 7461.119050] Lustre: server umount lustre-OST0001 complete
[ 7477.256115] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local
[ 7487.374915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 7487.378640] Lustre: Skipped 1 previous similar message
[ 7492.028124] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7494.879462] Lustre: 162307:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 7500.959863] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 7507.017402] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7512.166351] LustreError: 162672:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 7512.192398] LustreError: 162672:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message
[ 7512.225365] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:810 to 0x240000400:833)
[ 7517.047543] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 7522.313183] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:810 to 0x280000400:833)
[ 7523.724508] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7532.102557] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 7536.181557] Lustre: 164225:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 7605.706368] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 17:22:43 (1788470563)
[ 7682.289169] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 17:24:00 (1788470640)
[ 9487.867540] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 17:54:05 (1788472445)
[ 9506.592874] Lustre: server umount lustre-MDT0000 complete
[ 9511.562869] LustreError: 161859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788472470 with bad export cookie 17935766906339097447
[ 9511.585691] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 9521.262895] Lustre: server umount lustre-OST0000 complete
[ 9525.220678] Lustre: 3312:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788472468/real 1788472468]  req@ffff8ea0cc3fca80 x1875339466417152/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788472484 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 9525.240083] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 9526.019412] Lustre: server umount lustre-OST0001 complete
[ 9543.554499] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local
[ 9554.622908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 9560.630438] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9564.296327] Lustre: 172504:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 9572.294407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 9579.431629] LustreError: 172870:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9579.503217] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5835 to 0x240000400:5857)
[ 9579.998422] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9584.611338] LustreError: 172870:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9589.730635] LustreError: 173217:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9591.641510] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 9596.960705] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5835 to 0x280000400:5857)
[ 9599.433367] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9608.495451] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 9612.418973] Lustre: 174428:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 9651.949376] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 17:56:49 (1788472609)
[ 9703.634458] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 17:57:40 (1788472660)
[ 9752.926096] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 17:58:30 (1788472710)
[ 9754.991500] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs
[ 9757.316405] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 17:58:35 (1788472715)
[ 9759.208469] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs
[ 9761.479392] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 17:58:39 (1788472719)
[ 9859.597697] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 18:00:17 (1788472817)
[ 9915.698592] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 18:01:13 (1788472873)
[10016.380188] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 18:02:54 (1788472974)
[10029.503599] Lustre: *** cfs_fail_loc=a09, val=0***
[10029.512950] Lustre: Skipped 2 previous similar messages
[10031.542452] Lustre: *** cfs_fail_loc=a09, val=0***
[10031.544464] Lustre: Skipped 79 previous similar messages
[10035.555467] Lustre: *** cfs_fail_loc=a09, val=0***
[10035.560218] Lustre: Skipped 191 previous similar messages
[10043.577296] Lustre: *** cfs_fail_loc=a09, val=0***
[10043.581149] Lustre: Skipped 371 previous similar messages
[10059.579652] Lustre: *** cfs_fail_loc=a09, val=0***
[10059.582492] Lustre: Skipped 860 previous similar messages
[10091.594875] Lustre: *** cfs_fail_loc=a09, val=0***
[10091.596837] Lustre: Skipped 1576 previous similar messages
[10155.596091] Lustre: *** cfs_fail_loc=a09, val=0***
[10155.598213] Lustre: Skipped 3084 previous similar messages
[10563.501459] Lustre: *** cfs_fail_loc=a08, val=0***
[10563.516355] Lustre: Skipped 1840 previous similar messages
[10563.523229] Lustre: *** cfs_fail_loc=a08, val=0***
[10563.529535] Lustre: Skipped 4 previous similar messages
[10830.827857] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[10830.829845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[10830.835182] Lustre: Skipped 1 previous similar message
[10830.846907] Lustre: Skipped 1 previous similar message
[10831.583190] Lustre: server umount lustre-MDT0000 complete
[10835.723263] LustreError: 172057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788473794 with bad export cookie 17935766906340876203
[10835.736927] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[10835.970812] Lustre: server umount lustre-OST0000 complete
[10840.844591] Lustre: server umount lustre-OST0001 complete
[10849.558157] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid
[10860.228457] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local
[10871.901542]  vdc: vdc1 vdc9
[10883.884047]  vde: vde1 vde9
[10896.965164]  vdf: vdf1 vdf9
[10897.025636]  vdf: vdf1 vdf9
[10897.081862]  vdf: vdf1 vdf9
[10918.344244] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local
[10930.046617] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000
[10930.460119] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[10930.554525] Lustre: lustre-MDT0000: new disk, initializing
[10930.990586] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[10931.074602] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[10937.463881] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10944.103168] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params
[10953.350970] Lustre: lustre-OST0000: new disk, initializing
[10953.364334] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[10953.375793] Lustre: Skipped 1 previous similar message
[10953.721451] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[10954.951744] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost
[10954.962714] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost]
[10955.307953] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400
[10966.113484] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10983.578460] Lustre: lustre-OST0001: new disk, initializing
[10983.581658] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[10983.763691] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[10985.048200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost
[10985.053868] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost]
[10985.266262] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400
[10992.203275] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[11004.340412] Lustre: DEBUG MARKER: Using TIMEOUT=20
[11008.988171] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params
[11052.870047] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 18:20:10 (1788474010)
[11060.965626] Lustre: DEBUG MARKER: == sanity-quota test 49c: lfs quota long options don't consume an extra argument ========================================================== 18:20:19 (1788474019)
[11070.179838] Lustre: DEBUG MARKER: == sanity-quota test 49d: lfs quota -d and --delimiter both work ========================================================== 18:20:28 (1788474028)
[11079.035551] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 18:20:36 (1788474036)
[11131.652800] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 18:21:29 (1788474089)
[11205.240314] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 18:22:42 (1788474162)
[11243.676719] Lustre: DEBUG MARKER: rename directory return 255
[11301.000662] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 18:24:18 (1788474258)
[11338.098980] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 18:24:56 (1788474296)
[11390.124887] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 18:25:47 (1788474347)
[11523.909230] LustreError: 196358:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-0x0 id:60001 enforced:1 hard:51200 soft:0 granted:51200 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[11568.116996] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 18:28:46 (1788474526)
[11607.298639] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 18:29:25 (1788474565)
[11653.829374] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 18:30:11 (1788474611)
[11849.153952] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 18:33:27 (1788474807)
[11850.982747] Lustre: DEBUG MARKER: SKIP: sanity-quota test_59 ldiskfs only test
[11853.533970] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 18:33:30 (1788474810)
[11917.476955] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[11919.549223] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 18:34:37 (1788474877)
[11952.171284] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[11954.493261] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 18:35:12 (1788474912)
[12011.727443] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[12013.327919] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 18:36:11 (1788474971)
[12063.237714] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 18:37:00 (1788475020)
[12064.782100] Lustre: DEBUG MARKER: SKIP: sanity-quota test_67 ZFS grants some block space together with inode
[12067.066992] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 18:37:04 (1788475024)
[12094.637327] LustreError: 212753:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[12139.642428] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 18:38:17 (1788475097)
[12167.410200] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[12169.324393] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[12361.529398] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 18:41:58 (1788475318)
[12421.959865] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 18:42:59 (1788475379)
[12459.125674] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 18:43:37 (1788475417)
[12461.184786] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71a ZFS grants some block space together with inode
[12463.915743] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 18:43:41 (1788475421)
[12466.096548] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71b ZFS grants some block space together with inode
[12468.500967] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 18:43:46 (1788475426)
[12486.523141] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB)
[12506.834856] Lustre: DEBUG MARKER: Write...
[12510.348116] Lustre: DEBUG MARKER: Write out of block quota ...
[12575.775887] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 18:45:33 (1788475533)
[12605.404525] Lustre: DEBUG MARKER: set to use default quota
[12607.809945] Lustre: DEBUG MARKER: set default quota
[12609.376314] Lustre: DEBUG MARKER: get default quota
[12616.978106] Lustre: DEBUG MARKER: Test not out of quota
[12621.856145] Lustre: DEBUG MARKER: Test out of quota
[12638.438488] Lustre: DEBUG MARKER: Increase default quota
[12670.713562] Lustre: DEBUG MARKER: Set quota to override default quota
[12670.804729] LustreError: 189336:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45063 time:1789080430 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[12685.189239] Lustre: DEBUG MARKER: Set to use default quota again
[12710.972746] Lustre: DEBUG MARKER: Cleanup
[12794.099604] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 18:49:12 (1788475752)
[12822.385465] Lustre: DEBUG MARKER: set default quota for qpool1
[12824.425882] Lustre: DEBUG MARKER: Write from user that hasn't lqe
[12886.606303] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 18:50:43 (1788475843)
[12983.213321] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 18:52:21 (1788475941)
[13069.892170] Lustre: DEBUG MARKER: Write...
[13073.494862] Lustre: DEBUG MARKER: Write out of block quota ...
[13209.168543] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 18:56:06 (1788476166)
[13257.482934] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 18:56:55 (1788476215)
[13265.673898] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 18:57:04 (1788476224)
[13268.599649] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78A fallocate not supported
[13270.545748] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 18:57:08 (1788476228)
[13273.646225] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78a fallocate not supported
[13276.025150] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 18:57:13 (1788476233)
[13298.170976] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 18:57:35 (1788476255)
[13299.877647] Lustre: DEBUG MARKER: SKIP: sanity-quota test_80 ZFS grants some block space together with inode
[13302.262755] Lustre: DEBUG MARKER: == sanity-quota test 81: Race qmt_start_pool_recalc with qmt_pool_free ========================================================== 18:57:40 (1788476260)
[13319.730813] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[13326.141922] LustreError: 238028:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 sleeping for 10000ms
[13329.907980] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[13329.920245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[13329.938106] Lustre: Skipped 1 previous similar message
[13335.008674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[13335.014173] Lustre: Skipped 3 previous similar messages
[13336.175069] LustreError: 238028:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 awake
[13336.551387] Lustre: server umount lustre-MDT0000 complete
[13345.029352] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[13345.594213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[13345.808704] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:27 to 0x280000400:65)
[13345.821065] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:27 to 0x240000400:65)
[13350.842621] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[13360.098307] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[13360.108291] Lustre: Skipped 1 previous similar message
[13390.314875] Lustre: DEBUG MARKER: == sanity-quota test 82: verify more than 8 qids for single operation ========================================================== 18:59:08 (1788476348)
[13426.158643] Lustre: DEBUG MARKER: == sanity-quota test 83: Setting default quota shouldn't affect grace time ========================================================== 18:59:43 (1788476383)
[13458.283888] Lustre: DEBUG MARKER: == sanity-quota test 84: Reset quota should fix the insane granted quota ========================================================== 19:00:16 (1788476416)
[13513.248974] Lustre: *** cfs_fail_loc=a08, val=0***
[13513.256035] Lustre: Skipped 7 previous similar messages
[13513.276137] Lustre: *** cfs_fail_loc=a08, val=0***
[13513.281359] Lustre: Skipped 9 previous similar messages
[13632.466918] Lustre: DEBUG MARKER: == sanity-quota test 85: do not hung at write with the least_qunit ========================================================== 19:03:10 (1788476590)
[13719.935632] Lustre: DEBUG MARKER: == sanity-quota test 86: Pre-acquired quota should be released if quota is over limit ========================================================== 19:04:38 (1788476678)
[13792.895599] LustreError: 239319:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1789081552 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[13945.913111] LustreError: 243489:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1789081705 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[13955.163611] Lustre: DEBUG MARKER: sanity-quota test_86: @@@@@@ FAIL: succeed to create files, expect failed