[ 5993.254679] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 6001.127042] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 6005.488985] LustreError: 125011:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 6006.858492] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 6008.950463] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 6011.873494] Lustre: Failing over lustre-MDT0000
[ 6012.409241] Lustre: server umount lustre-MDT0000 complete
[ 6032.447659] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6032.847319] Lustre: 3310:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788644444/real 1788644444]  req@ffff8f83eebeb800 x1875523421885696/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788644460 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6032.879490] Lustre: 3310:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 6032.888701] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6033.119552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6033.175430] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 6033.951158] Lustre: 3312:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788644444/real 1788644444]  req@ffff8f83eebea680 x1875523421885568/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788644460 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6037.088795] Lustre: 3312:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788644449/real 1788644449]  req@ffff8f83facc6680 x1875523421885952/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788644465 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6038.506735] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 6038.798289] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6039.519216] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 6039.713819] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 6039.793840] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:634 to 0x240000400:673)
[ 6039.801680] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:631 to 0x280000400:673)
[ 6042.015641] Lustre: 3311:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788644454/real 1788644454]  req@ffff8f8306adc000 x1875523421886464/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788644470 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6042.064982] Lustre: 3311:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 6049.597485] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6051.495355] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6057.946085] Lustre: DEBUG MARKER: (dd_pid=117592, time=0, timeout=600)
[ 6106.063836] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 6113.236896] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 6116.729549] LustreError: 127874:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 6117.634394] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 6119.697801] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 6122.101994] Lustre: Failing over lustre-MDT0000
[ 6122.535618] Lustre: server umount lustre-MDT0000 complete
[ 6138.849406] Lustre: 3309:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788644550/real 1788644550]  req@ffff8f83faf0e680 x1875523421924480/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1788644566 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 6138.874294] Lustre: 3309:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 6138.887380] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6138.900660] Lustre: Skipped 1 previous similar message
[ 6142.047227] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6153.000307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6153.187922] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 6157.197758] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6163.172216] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[ 6163.181822] Lustre: Skipped 1 previous similar message
[ 6166.480688] LustreError: 128527:0:(tgt_handler.c:534:tgt_filter_recovery_request()) @@@ not permitted during recovery  req@ffff8f830e6df800 x1875523421939072/t0(0) o601->lustre-MDT0000-lwp-OST0000_UUID@0@lo:530/0 lens 336/0 e 0 to 0 dl 1788644605 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'qsd_reint_0.lus.0' uid:0 gid:0 projid:4294967295
[ 6166.496070] LustreError: lustre-MDT0000-lwp-OST0000: operation quota_acquire to node 0@lo failed: rc = -11
[ 6166.501390] LustreError: Skipped 1 previous similar message
[ 6167.516601] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 6167.652318] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 6167.726889] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:675 to 0x240000400:705)
[ 6167.738159] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:631 to 0x280000400:705)
[ 6174.971958] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6177.320720] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6183.732132] Lustre: DEBUG MARKER: (dd_pid=119979, time=0, timeout=600)
[ 6255.289815] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 17:44:41 (1788644681)
[ 6278.208750] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 6280.129553] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 6286.922995] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 6288.619698] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 6323.363882] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 17:45:49 (1788644749)
[ 6352.184309] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 6370.461279] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 6372.772703] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 6375.001679] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 6377.521652] Lustre: DEBUG MARKER: Set quota for 1 times
[ 6381.392390] Lustre: DEBUG MARKER: Set quota for 2 times
[ 6385.059513] Lustre: DEBUG MARKER: Set quota for 3 times
[ 6388.469724] Lustre: DEBUG MARKER: Set quota for 4 times
[ 6392.362510] Lustre: DEBUG MARKER: Set quota for 5 times
[ 6396.731585] Lustre: DEBUG MARKER: Set quota for 6 times
[ 6400.306901] Lustre: DEBUG MARKER: Set quota for 7 times
[ 6404.357898] Lustre: DEBUG MARKER: Set quota for 8 times
[ 6447.546425] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 17:47:54 (1788644874)
[ 6462.190263] Lustre: server umount lustre-MDT0000 complete
[ 6466.825289] LustreError: 28156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788644894 with bad export cookie 4771532827458547191
[ 6466.835237] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6476.369771] Lustre: server umount lustre-OST0000 complete
[ 6480.800578] Lustre: 3312:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788644892/real 1788644892]  req@ffff8f8400f69c00 x1875523422049536/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788644908 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6480.862882] Lustre: 3312:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
[ 6480.874087] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6480.891377] Lustre: Skipped 1 previous similar message
[ 6481.372839] Lustre: server umount lustre-OST0001 complete
[ 6497.990312] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local
[ 6508.406489] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6513.061915] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6516.388564] Lustre: 137756:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6522.882081] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 6529.725700] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6533.097596] LustreError: 138121:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6533.112504] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:708 to 0x240000400:737)
[ 6538.166990] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6543.381554] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:707 to 0x280000400:737)
[ 6544.375780] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6552.734251] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6557.581259] Lustre: 139669:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6578.271765] Lustre: server umount lustre-MDT0000 complete
[ 6583.916775] LustreError: 137301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788645011 with bad export cookie 4771532827458556942
[ 6583.922223] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6593.600847] Lustre: server umount lustre-OST0000 complete
[ 6595.551087] Lustre: 3312:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788645007/real 1788645007]  req@ffff8f830de85f80 x1875523422091008/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788645023 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6598.074764] Lustre: server umount lustre-OST0001 complete
[ 6615.584856] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local
[ 6628.458980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6633.899977] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6637.411743] Lustre: 142328:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6650.892174] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6653.866466] LustreError: 142707:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6653.899750] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:708 to 0x240000400:769)
[ 6660.505278] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6660.511296] Lustre: Skipped 1 previous similar message
[ 6665.765115] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:707 to 0x280000400:769)
[ 6667.196614] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6676.377412] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6680.857958] Lustre: 144256:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6692.828359] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 17:51:59 (1788645119)
[ 6694.652555] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 6696.561860] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 17:52:03 (1788645123)
[ 6753.698969] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 17:53:00 (1788645180)
[ 6811.057186] Lustre: DEBUG MARKER: Write...
[ 6814.404760] Lustre: DEBUG MARKER: Write out of block quota ...
[ 6871.160278] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 17:54:57 (1788645297)
[ 6878.997525] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 17:55:05 (1788645305)
[ 6891.094642] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 17:55:17 (1788645317)
[ 6901.153846] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 17:55:27 (1788645327)
[ 6909.674331] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 17:55:36 (1788645336)
[ 6972.095935] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 7147.911254] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 7336.810192] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 18:02:43 (1788645763)
[ 7409.180747] Lustre: DEBUG MARKER: Restart...
[ 7413.600964] Lustre: server umount lustre-MDT0000 complete
[ 7417.036551] LustreError: 141882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788645845 with bad export cookie 4771532827458558475
[ 7417.050565] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 7427.631057] Lustre: server umount lustre-OST0000 complete
[ 7442.888843] Lustre: server umount lustre-OST0001 complete
[ 7461.927503] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local
[ 7472.462413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 7476.764207] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7479.761261] Lustre: 162559:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 7487.074240] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 7493.580624] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7497.314587] LustreError: 162924:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 7497.341951] LustreError: 162924:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message
[ 7497.342392] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:778 to 0x240000400:801)
[ 7501.952678] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 7506.938087] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:778 to 0x280000400:801)
[ 7509.485823] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7518.933853] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 7524.164777] Lustre: 164467:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 7596.892124] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 18:07:03 (1788646023)
[ 7683.518813] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 18:08:30 (1788646110)
[ 9533.415261] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 18:39:20 (1788647960)
[ 9552.214628] Lustre: server umount lustre-MDT0000 complete
[ 9558.443400] LustreError: 162112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788647986 with bad export cookie 4771532827458564964
[ 9558.468757] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 9568.239701] Lustre: 3312:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788647980/real 1788647980]  req@ffff8f82ca70b480 x1875523425357824/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788647996 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 9568.309357] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 9568.355218] Lustre: Skipped 1 previous similar message
[ 9569.624216] Lustre: server umount lustre-OST0000 complete
[ 9576.549298] Lustre: server umount lustre-OST0001 complete
[ 9597.758387] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local
[ 9608.285771] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 9613.082085] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9616.770677] Lustre: 173025:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 9624.914345] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 9633.002627] LustreError: 173407:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9633.086595] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5803 to 0x240000400:5825)
[ 9636.595945] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9638.391687] LustreError: 173647:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9643.491302] LustreError: 173406:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9646.032069] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 9651.216756] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5803 to 0x280000400:5825)
[ 9653.565912] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9661.935264] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 9665.883744] Lustre: 174959:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 9705.526966] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 18:42:12 (1788648132)
[ 9753.557298] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 18:42:59 (1788648179)
[ 9801.216190] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 18:43:47 (1788648227)
[ 9803.146832] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs
[ 9805.432362] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 18:43:51 (1788648231)
[ 9808.559770] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs
[ 9810.883408] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 18:43:57 (1788648237)
[ 9902.267443] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 18:45:28 (1788648328)
[ 9954.127408] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 18:46:21 (1788648381)
[10039.766325] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 18:47:46 (1788648466)
[10050.745745] Lustre: *** cfs_fail_loc=a09, val=0***
[10050.753186] Lustre: Skipped 2 previous similar messages
[10052.748896] Lustre: *** cfs_fail_loc=a09, val=0***
[10052.754604] Lustre: Skipped 97 previous similar messages
[10056.754393] Lustre: *** cfs_fail_loc=a09, val=0***
[10056.758165] Lustre: Skipped 161 previous similar messages
[10064.773205] Lustre: *** cfs_fail_loc=a09, val=0***
[10064.783313] Lustre: Skipped 331 previous similar messages
[10080.798797] Lustre: *** cfs_fail_loc=a09, val=0***
[10080.803449] Lustre: Skipped 647 previous similar messages
[10112.828897] Lustre: *** cfs_fail_loc=a09, val=0***
[10112.830849] Lustre: Skipped 1479 previous similar messages
[10176.838526] Lustre: *** cfs_fail_loc=a09, val=0***
[10176.843392] Lustre: Skipped 2625 previous similar messages
[10583.778611] Lustre: *** cfs_fail_loc=a08, val=0***
[10583.783455] Lustre: Skipped 2657 previous similar messages
[10583.790588] Lustre: *** cfs_fail_loc=a08, val=0***
[10583.795035] Lustre: Skipped 1 previous similar message
[10879.969271] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[10879.981150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[10879.987687] Lustre: Skipped 1 previous similar message
[10883.998059] Lustre: server umount lustre-MDT0000 complete
[10888.196628] LustreError: 181623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788649316 with bad export cookie 4771532827460345344
[10888.209489] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[10888.596637] Lustre: server umount lustre-OST0000 complete
[10893.315660] Lustre: server umount lustre-OST0001 complete
[10901.364871] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid
[10907.905996] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local
[10919.227844]  vdc: vdc1 vdc9
[10929.109188]  vde: vde1 vde9
[10940.480377]  vdf: vdf1 vdf9
[10940.519891]  vdf: vdf1 vdf9
[10940.556660]  vdf: vdf1 vdf9
[10958.240979] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local
[10968.008942] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000
[10968.226959] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[10968.315640] Lustre: lustre-MDT0000: new disk, initializing
[10968.690545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[10968.747959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[10973.388132] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10978.331852] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params
[10986.614887] Lustre: lustre-OST0000: new disk, initializing
[10986.616889] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[10986.619790] Lustre: Skipped 1 previous similar message
[10986.694225] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[10988.515812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost
[10988.530198] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost]
[10988.643048] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400
[10993.544641] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[11003.533415] Lustre: lustre-OST0001: new disk, initializing
[11003.546558] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[11003.707072] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[11005.235130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost
[11005.257789] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost]
[11005.665739] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400
[11013.204840] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[11024.860510] Lustre: DEBUG MARKER: Using TIMEOUT=20
[11029.274936] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params
[11068.019382] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 19:04:54 (1788649494)
[11076.435675] Lustre: DEBUG MARKER: == sanity-quota test 49c: lfs quota long options don't consume an extra argument ========================================================== 19:05:03 (1788649503)
[11085.167900] Lustre: DEBUG MARKER: == sanity-quota test 49d: lfs quota -d and --delimiter both work ========================================================== 19:05:11 (1788649511)
[11093.055793] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 19:05:19 (1788649519)
[11145.609693] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 19:06:12 (1788649572)
[11206.637943] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 19:07:13 (1788649633)
[11239.295898] Lustre: DEBUG MARKER: rename directory return 255
[11288.277339] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 19:08:34 (1788649714)
[11323.744592] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 19:09:10 (1788649750)
[11370.056742] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 19:09:56 (1788649796)
[11501.880570] LustreError: 200386:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-0x0 id:60001 enforced:1 hard:51200 soft:0 granted:51200 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[11551.803875] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 19:12:58 (1788649978)
[11592.584920] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 19:13:39 (1788650019)
[11640.293880] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 19:14:26 (1788650066)
[11853.878426] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 19:18:00 (1788650280)
[11855.717579] Lustre: DEBUG MARKER: SKIP: sanity-quota test_59 ldiskfs only test
[11858.265808] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 19:18:04 (1788650284)
[11926.633428] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[11928.697171] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 19:19:15 (1788650355)
[11964.760696] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[11966.624572] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 19:19:53 (1788650393)
[12020.769690] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[12022.430670] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 19:20:49 (1788650449)
[12074.456506] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 19:21:40 (1788650500)
[12076.519161] Lustre: DEBUG MARKER: SKIP: sanity-quota test_67 ZFS grants some block space together with inode
[12078.729463] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 19:21:45 (1788650505)
[12109.365112] LustreError: 213161:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[12158.524575] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 19:23:05 (1788650585)
[12187.899520] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[12189.853084] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[12405.244924] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 19:27:12 (1788650832)
[12461.120521] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 19:28:08 (1788650888)
[12498.542574] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 19:28:45 (1788650925)
[12500.224852] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71a ZFS grants some block space together with inode
[12502.433443] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 19:28:49 (1788650929)
[12504.459165] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71b ZFS grants some block space together with inode
[12506.734304] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 19:28:53 (1788650933)
[12524.182937] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB)
[12541.362615] Lustre: DEBUG MARKER: Write...
[12544.438634] Lustre: DEBUG MARKER: Write out of block quota ...
[12606.279495] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 19:30:32 (1788651032)
[12635.692319] Lustre: DEBUG MARKER: set to use default quota
[12637.440355] Lustre: DEBUG MARKER: set default quota
[12639.451127] Lustre: DEBUG MARKER: get default quota
[12645.467817] Lustre: DEBUG MARKER: Test not out of quota
[12650.002451] Lustre: DEBUG MARKER: Test out of quota
[12663.777368] Lustre: DEBUG MARKER: Increase default quota
[12698.995802] Lustre: DEBUG MARKER: Set quota to override default quota
[12699.061640] LustreError: 189884:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45063 time:1789255927 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[12711.955278] Lustre: DEBUG MARKER: Set to use default quota again
[12735.221854] Lustre: DEBUG MARKER: Cleanup
[12806.216589] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 19:33:53 (1788651233)
[12832.540286] Lustre: DEBUG MARKER: set default quota for qpool1
[12834.501849] Lustre: DEBUG MARKER: Write from user that hasn't lqe
[12890.864299] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 19:35:17 (1788651317)
[12982.723790] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 19:36:49 (1788651409)
[13062.669451] Lustre: DEBUG MARKER: Write...
[13066.531734] Lustre: DEBUG MARKER: Write out of block quota ...
[13212.052087] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 19:40:38 (1788651638)
[13260.774676] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 19:41:27 (1788651687)
[13269.953673] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 19:41:36 (1788651696)
[13273.350566] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78A fallocate not supported
[13276.163122] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 19:41:42 (1788651702)
[13279.718538] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78a fallocate not supported
[13282.332822] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 19:41:48 (1788651708)
[13306.887983] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 19:42:13 (1788651733)
[13308.955095] Lustre: DEBUG MARKER: SKIP: sanity-quota test_80 ZFS grants some block space together with inode
[13311.388343] Lustre: DEBUG MARKER: == sanity-quota test 81: Race qmt_start_pool_recalc with qmt_pool_free ========================================================== 19:42:17 (1788651737)
[13330.764336] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[13338.644431] LustreError: 238327:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 sleeping for 10000ms
[13343.745712] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.57@tcp (stopping)
[13344.736108] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[13344.738750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[13344.764121] Lustre: Skipped 2 previous similar messages
[13344.785226] Lustre: Skipped 1 previous similar message
[13348.679171] LustreError: 238327:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 awake
[13348.829525] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.57@tcp (stopping)
[13348.995032] Lustre: server umount lustre-MDT0000 complete
[13358.426204] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[13359.223721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[13359.381461] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:27 to 0x240000400:65)
[13359.382726] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:27 to 0x280000400:65)
[13364.223321] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[13364.234428] Lustre: Skipped 1 previous similar message
[13364.495751] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[13406.782129] Lustre: DEBUG MARKER: == sanity-quota test 82: verify more than 8 qids for single operation ========================================================== 19:43:53 (1788651833)
[13446.104401] Lustre: DEBUG MARKER: == sanity-quota test 83: Setting default quota shouldn't affect grace time ========================================================== 19:44:32 (1788651872)
[13484.775921] Lustre: DEBUG MARKER: == sanity-quota test 84: Reset quota should fix the insane granted quota ========================================================== 19:45:11 (1788651911)
[13547.564791] Lustre: *** cfs_fail_loc=a08, val=0***
[13547.570406] Lustre: Skipped 6 previous similar messages
[13547.577051] Lustre: *** cfs_fail_loc=a08, val=0***
[13547.583425] Lustre: Skipped 15 previous similar messages
[13689.138977] Lustre: DEBUG MARKER: == sanity-quota test 85: do not hung at write with the least_qunit ========================================================== 19:48:35 (1788652115)
[13776.465207] Lustre: DEBUG MARKER: == sanity-quota test 86: Pre-acquired quota should be released if quota is over limit ========================================================== 19:50:02 (1788652202)
[13865.033782] LustreError: 238915:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1789257093 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[13874.160726] Lustre: DEBUG MARKER: sanity-quota test_86: @@@@@@ FAIL: succeed to create files, expect failed