[ 5183.917435] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5189.510337] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 5193.531268] LustreError: 121241:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5194.928429] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5196.920369] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5198.835683] Lustre: Failing over lustre-MDT0000
[ 5199.254104] Lustre: server umount lustre-MDT0000 complete
[ 5216.225384] Lustre: 3308:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787390299/real 1787390299]  req@ffff9a27fac0c380 x1874209186025728/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1787390315 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 5216.274781] Lustre: 3308:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5216.282049] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5216.734174] LustreError: MGC192.168.206.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5217.436675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5217.488251] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5218.428753] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5218.473348] Lustre: 3307:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787390300/real 1787390300]  req@ffff9a270da5ca80 x1874209186026112/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787390316 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5218.641348] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5218.697716] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:676 to 0x280000400:705)
[ 5218.703151] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:681 to 0x240000400:705)
[ 5221.419426] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 5221.870356] Lustre: DEBUG MARKER: oleg623-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5223.392145] Lustre: 3306:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787390306/real 1787390306]  req@ffff9a27f59bbb80 x1874209186026496/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787390322 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5223.436211] Lustre: 3306:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5228.514357] Lustre: 3309:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787390311/real 1787390311]  req@ffff9a27fac0dc00 x1874209186026752/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787390327 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5228.539632] Lustre: 3309:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5230.582431] Lustre: DEBUG MARKER: oleg623-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5232.155849] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5245.796461] Lustre: DEBUG MARKER: (dd_pid=114675, time=8, timeout=600)
[ 5295.589853] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5301.287225] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 5304.126438] LustreError: 124163:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5304.822424] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5306.029511] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5307.595376] Lustre: Failing over lustre-MDT0000
[ 5307.833490] LustreError: lustre-MDT0000-lwp-OST0000: operation quota_acquire to node 0@lo failed: rc = -107
[ 5307.839759] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5307.866393] Lustre: Skipped 1 previous similar message
[ 5307.882195] LustreError: 122608:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5307.899128] LustreError: 122608:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message
[ 5308.007267] Lustre: server umount lustre-MDT0000 complete
[ 5324.772284] Lustre: 3307:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787390407/real 1787390407]  req@ffff9a26c6af8000 x1874209186065920/t0(0) o400->MGC192.168.206.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787390423 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5324.796175] Lustre: 3307:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5324.806000] LustreError: MGC192.168.206.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5335.150718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5335.349179] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5335.971962] LustreError: 3305:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-lwp-OST0001: namespace resource [0x200000006:0x20000:0x0].0x0 (ffff9a27fae1d100) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5335.998187] LustreError: 3305:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 5 previous similar messages
[ 5336.019727] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 5336.032118] Lustre: Skipped 1 previous similar message
[ 5336.669731] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5336.809094] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5336.868184] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:707 to 0x240000400:737)
[ 5336.870571] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:676 to 0x280000400:737)
[ 5340.717423] Lustre: DEBUG MARKER: oleg623-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5341.157315] Lustre: 3308:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787390423/real 1787390423]  req@ffff9a26c6af8e00 x1874209186068736/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787390439 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5341.215871] Lustre: 3308:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
[ 5351.961555] Lustre: DEBUG MARKER: oleg623-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5354.463416] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5365.683325] Lustre: DEBUG MARKER: (dd_pid=117169, time=4, timeout=600)
[ 5424.403829] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 05:22:02 (1787390522)
[ 5444.971291] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 5446.558285] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 5452.131738] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 5453.642339] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 5488.113172] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 05:23:05 (1787390585)
[ 5509.871133] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 5525.190386] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 5526.617220] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 5528.441952] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 5530.447379] Lustre: DEBUG MARKER: Set quota for 1 times
[ 5534.098380] Lustre: DEBUG MARKER: Set quota for 2 times
[ 5537.359954] Lustre: DEBUG MARKER: Set quota for 3 times
[ 5540.720839] Lustre: DEBUG MARKER: Set quota for 4 times
[ 5544.476805] Lustre: DEBUG MARKER: Set quota for 5 times
[ 5548.410928] Lustre: DEBUG MARKER: Set quota for 6 times
[ 5552.970783] Lustre: DEBUG MARKER: Set quota for 7 times
[ 5556.953548] Lustre: DEBUG MARKER: Set quota for 8 times
[ 5614.499396] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 05:25:12 (1787390712)
[ 5628.275650] Lustre: server umount lustre-MDT0000 complete
[ 5631.859260] LustreError: 5780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390730 with bad export cookie 1984290552929166967
[ 5631.868779] LustreError: MGC192.168.206.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5641.256431] Lustre: server umount lustre-OST0000 complete
[ 5655.115285] Lustre: server umount lustre-OST0001 complete
[ 5668.481295] Lustre: DEBUG MARKER: oleg623-server.virtnet: executing load_modules_local
[ 5677.020476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5681.096908] Lustre: DEBUG MARKER: oleg623-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5683.895956] Lustre: 133724:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 5688.842068] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 5694.949641] Lustre: DEBUG MARKER: oleg623-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5701.089399] LustreError: 134039:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5701.125385] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:739 to 0x240000400:769)
[ 5701.536735] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 5706.745520] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:740 to 0x280000400:769)
[ 5707.081373] Lustre: DEBUG MARKER: oleg623-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5713.364836] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 5716.107670] Lustre: 135549:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 5731.545737] Lustre: server umount lustre-MDT0000 complete
[ 5735.260881] LustreError: 133271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390834 with bad export cookie 1984290552929173953
[ 5735.268751] LustreError: MGC192.168.206.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5745.732309] Lustre: server umount lustre-OST0000 complete
[ 5748.706561] Lustre: 3306:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787390831/real 1787390831]  req@ffff9a27c858ca80 x1874209186228608/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787390847 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5748.728881] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5748.738026] Lustre: Skipped 1 previous similar message
[ 5749.069759] Lustre: server umount lustre-OST0001 complete
[ 5761.844641] Lustre: DEBUG MARKER: oleg623-server.virtnet: executing load_modules_local
[ 5769.648161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5773.339397] Lustre: DEBUG MARKER: oleg623-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5776.014675] Lustre: 138196:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 5785.642615] Lustre: DEBUG MARKER: oleg623-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5792.016311] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 5792.021166] Lustre: Skipped 1 previous similar message
[ 5795.088542] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:739 to 0x240000400:801)
[ 5795.093341] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:740 to 0x280000400:801)
[ 5797.428064] Lustre: DEBUG MARKER: oleg623-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5803.144717] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 5805.800211] Lustre: 139996:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 5814.363608] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 05:28:32 (1787390912)
[ 5815.536686] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 5816.937250] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 05:28:34 (1787390914)
[ 5866.915272] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 05:29:24 (1787390964)
[ 5911.966437] Lustre: DEBUG MARKER: Write...
[ 5914.203351] Lustre: DEBUG MARKER: Write out of block quota ...
[ 5972.641816] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 05:31:10 (1787391070)
[ 5978.442696] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 05:31:15 (1787391075)
[ 5988.256431] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 05:31:25 (1787391085)
[ 5997.516860] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 05:31:35 (1787391095)
[ 6004.917281] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 05:31:42 (1787391102)
[ 6070.733480] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 6211.895525] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 6401.748470] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 05:38:19 (1787391499)
[ 6464.462516] Lustre: DEBUG MARKER: Restart...
[ 6471.138447] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6471.160416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6471.174579] Lustre: Skipped 1 previous similar message
[ 6475.302971] Lustre: server umount lustre-MDT0000 complete
[ 6479.441544] LustreError: 137750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391578 with bad export cookie 1984290552929175486
[ 6479.445639] LustreError: MGC192.168.206.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6479.460375] LustreError: 137750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 6479.561027] Lustre: server umount lustre-OST0000 complete
[ 6483.343821] Lustre: server umount lustre-OST0001 complete
[ 6497.273627] Lustre: DEBUG MARKER: oleg623-server.virtnet: executing load_modules_local
[ 6506.924241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6511.307321] Lustre: DEBUG MARKER: oleg623-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6514.163161] Lustre: 158790:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6519.653402] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 6526.823269] Lustre: DEBUG MARKER: oleg623-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6531.874298] LustreError: 159116:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6531.934706] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:810 to 0x240000400:833)
[ 6534.359797] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6539.763694] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:810 to 0x280000400:833)
[ 6540.908248] Lustre: DEBUG MARKER: oleg623-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6550.040219] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6555.062286] Lustre: 160618:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6623.672904] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 05:42:01 (1787391721)
[ 6700.441236] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 05:43:17 (1787391797)
[ 8232.530769] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 06:08:50 (1787393330)
[ 8249.825458] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 8249.828136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 8249.855785] Lustre: Skipped 2 previous similar messages
[ 8249.887702] Lustre: Skipped 1 previous similar message
[ 8250.816146] Lustre: server umount lustre-MDT0000 complete
[ 8254.780539] LustreError: 158343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393353 with bad export cookie 1984290552929181982
[ 8254.784661] LustreError: MGC192.168.206.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 8254.804256] LustreError: 158343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 8254.894447] Lustre: server umount lustre-OST0000 complete
[ 8259.435835] Lustre: server umount lustre-OST0001 complete
[ 8277.758635] Lustre: DEBUG MARKER: oleg623-server.virtnet: executing load_modules_local
[ 8289.495304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 8294.420645] Lustre: DEBUG MARKER: oleg623-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8297.697157] Lustre: 169000:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 8306.637917] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 8314.787820] LustreError: 169318:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 8314.824775] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5835 to 0x240000400:5857)
[ 8315.481413] Lustre: DEBUG MARKER: oleg623-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8319.971289] LustreError: 169318:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 8325.098744] LustreError: 169319:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 8325.442901] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 8330.749283] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5835 to 0x280000400:5857)
[ 8332.433141] Lustre: DEBUG MARKER: oleg623-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8341.401198] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 8345.651070] Lustre: 170826:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 8382.229794] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 06:11:19 (1787393479)
[ 8428.273435] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 06:12:06 (1787393526)
[ 8475.764920] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 06:12:53 (1787393573)
[ 8477.846743] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs
[ 8480.127537] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 06:12:57 (1787393577)
[ 8481.565359] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs
[ 8483.135207] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 06:13:01 (1787393581)
[ 8578.041421] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 06:14:35 (1787393675)
[ 8591.043935] LustreError: 168601:0:(qsd_reint.c:633:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:1 until pending updates are flushed.
[ 8632.458185] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 06:15:29 (1787393729)
[ 8717.120528] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 06:16:54 (1787393814)
[ 8727.959837] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8727.961740] Lustre: Skipped 2 previous similar messages
[ 8728.993621] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8728.996398] Lustre: Skipped 45 previous similar messages
[ 8731.044869] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8731.048275] Lustre: Skipped 97 previous similar messages
[ 8735.065649] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8735.076250] Lustre: Skipped 223 previous similar messages
[ 8743.066153] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8743.075044] Lustre: Skipped 451 previous similar messages
[ 8759.070447] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8759.074817] Lustre: Skipped 905 previous similar messages
[ 8791.099744] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8791.103690] Lustre: Skipped 1749 previous similar messages
[ 8855.130205] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8855.133904] Lustre: Skipped 3529 previous similar messages
[ 9114.977363] Lustre: *** cfs_fail_loc=a08, val=0***
[ 9114.980167] Lustre: *** cfs_fail_loc=a08, val=0***
[ 9114.984671] Lustre: Skipped 995 previous similar messages
[ 9115.005930] Lustre: Skipped 3 previous similar messages
[ 9266.547960] Lustre: *** cfs_fail_loc=a08, val=0***
[ 9266.556577] Lustre: Skipped 14 previous similar messages
[ 9411.042843] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 9411.060253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 9415.478546] Lustre: server umount lustre-MDT0000 complete
[ 9419.685381] LustreError: 174398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394518 with bad export cookie 1984290552930956643
[ 9419.701653] LustreError: MGC192.168.206.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 9420.125655] Lustre: server umount lustre-OST0000 complete
[ 9424.794455] Lustre: server umount lustre-OST0001 complete
[ 9433.054690] Lustre: DEBUG MARKER: oleg623-server.virtnet: executing set_hostid
[ 9441.466200] Lustre: DEBUG MARKER: oleg623-server.virtnet: executing load_modules_local
[ 9451.425620]  vdc: vdc1 vdc9
[ 9451.443598]  vdc: vdc1 vdc9
[ 9451.463174]  vdc: vdc1 vdc9
[ 9462.939869]  vde: vde1 vde9
[ 9462.975087]  vde: vde1 vde9
[ 9463.023129]  vde: vde1 vde9
[ 9475.430240]  vdf: vdf1 vdf9
[ 9492.309931] Lustre: DEBUG MARKER: oleg623-server.virtnet: executing load_modules_local
[ 9501.740906] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000
[ 9501.999544] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[ 9502.088381] Lustre: lustre-MDT0000: new disk, initializing
[ 9502.655051] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 9502.767793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[ 9506.980199] Lustre: DEBUG MARKER: oleg623-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9511.780256] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params
[ 9517.115170] Lustre: lustre-OST0000: new disk, initializing
[ 9517.118887] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[ 9517.125222] Lustre: Skipped 1 previous similar message
[ 9517.214077] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 9518.386164] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost
[ 9518.392466] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost]
[ 9518.501355] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400
[ 9523.072385] Lustre: DEBUG MARKER: oleg623-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9531.952862] Lustre: lustre-OST0001: new disk, initializing
[ 9531.960499] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[ 9532.066067] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 9533.230129] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost
[ 9533.237827] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost]
[ 9533.379550] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400
[ 9538.311745] Lustre: DEBUG MARKER: oleg623-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9546.985356] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 9550.470767] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params
[ 9585.895963] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 06:31:23 (1787394683)
[ 9592.708723] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 06:31:30 (1787394690)
[ 9639.071455] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 06:32:16 (1787394736)
[ 9703.165564] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 06:33:20 (1787394800)
[ 9733.674463] Lustre: DEBUG MARKER: rename directory return 255
[ 9781.104374] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 06:34:38 (1787394878)
[ 9816.787177] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 06:35:14 (1787394914)
[ 9860.796296] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 06:35:58 (1787394958)
[10009.979291] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 06:38:27 (1787395107)
[10049.207308] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 06:39:06 (1787395146)
[10103.210513] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 06:40:00 (1787395200)
[10298.043992] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 06:43:15 (1787395395)
[10299.681616] Lustre: DEBUG MARKER: SKIP: sanity-quota test_59 ldiskfs only test
[10301.562936] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 06:43:19 (1787395399)
[10365.100440] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[10367.421651] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 06:44:24 (1787395464)
[10401.217580] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[10402.976101] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 06:45:00 (1787395500)
[10460.695231] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[10462.645786] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 06:46:00 (1787395560)
[10512.128329] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 06:46:49 (1787395609)
[10513.811515] Lustre: DEBUG MARKER: SKIP: sanity-quota test_67 ZFS grants some block space together with inode
[10515.639725] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 06:46:53 (1787395613)
[10546.969314] LustreError: 209038:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[10593.661953] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 06:48:11 (1787395691)
[10618.692192] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[10620.318133] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[10787.453344] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 06:51:24 (1787395884)
[10840.569695] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 06:52:18 (1787395938)
[10876.996276] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 06:52:54 (1787395974)
[10878.438406] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71a ZFS grants some block space together with inode
[10880.056407] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 06:52:57 (1787395977)
[10882.495858] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71b ZFS grants some block space together with inode
[10884.435914] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 06:53:01 (1787395981)
[10903.502205] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB)
[10919.755773] Lustre: DEBUG MARKER: Write...
[10922.765982] Lustre: DEBUG MARKER: Write out of block quota ...
[10978.455686] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 06:54:35 (1787396075)
[11005.931732] Lustre: DEBUG MARKER: set to use default quota
[11007.638567] Lustre: DEBUG MARKER: set default quota
[11009.223878] Lustre: DEBUG MARKER: get default quota
[11015.648201] Lustre: DEBUG MARKER: Test not out of quota
[11019.536714] Lustre: DEBUG MARKER: Test out of quota
[11031.858808] Lustre: DEBUG MARKER: Increase default quota
[11062.767971] Lustre: DEBUG MARKER: Set quota to override default quota
[11062.842726] LustreError: 185861:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45063 time:1788000961 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[11075.434675] Lustre: DEBUG MARKER: Set to use default quota again
[11097.662464] Lustre: DEBUG MARKER: Cleanup
[11171.512641] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 06:57:49 (1787396269)
[11196.632606] Lustre: DEBUG MARKER: set default quota for qpool1
[11198.349428] Lustre: DEBUG MARKER: Write from user that hasn't lqe
[11249.186201] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 06:59:06 (1787396346)
[11331.274437] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 07:00:29 (1787396429)
[11412.027895] Lustre: DEBUG MARKER: Write...
[11415.729976] Lustre: DEBUG MARKER: Write out of block quota ...
[11531.793390] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 07:03:49 (1787396629)
[11577.798794] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 07:04:35 (1787396675)
[11585.214399] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 07:04:42 (1787396682)
[11586.844639] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78A need >= 2.13.57 and ldiskfs for fallocate
[11588.802457] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 07:04:46 (1787396686)
[11590.610195] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78a need >= 2.13.57 and ldiskfs for fallocate
[11592.376084] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 07:04:50 (1787396690)
[11611.989986] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 07:05:09 (1787396709)
[11613.550365] Lustre: DEBUG MARKER: SKIP: sanity-quota test_80 ZFS grants some block space together with inode
[11615.219510] Lustre: DEBUG MARKER: == sanity-quota test 81: Race qmt_start_pool_recalc with qmt_pool_free ========================================================== 07:05:12 (1787396712)
[11632.063337] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[11638.884867] LustreError: 234107:0:(qmt_pool.c:1413:qmt_pool_recalc()) cfs_fail_timeout id a07 sleeping for 10000ms
[11642.851324] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[11642.855709] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[11642.871081] Lustre: Skipped 2 previous similar messages
[11642.876890] Lustre: Skipped 1 previous similar message
[11645.087841] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.23@tcp (stopping)
[11645.098549] Lustre: Skipped 1 previous similar message
[11647.970110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[11647.975970] Lustre: Skipped 1 previous similar message
[11648.968129] LustreError: 234107:0:(qmt_pool.c:1413:qmt_pool_recalc()) cfs_fail_timeout id a07 awake
[11649.476616] Lustre: server umount lustre-MDT0000 complete
[11656.917651] LustreError: MGC192.168.206.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[11657.453310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[11657.576627] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:27 to 0x240000400:65)
[11657.597943] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:27 to 0x280000400:65)
[11662.244524] Lustre: DEBUG MARKER: oleg623-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[11673.794570] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[11673.817182] Lustre: Skipped 1 previous similar message
[11701.355556] Lustre: DEBUG MARKER: == sanity-quota test 82: verify more than 8 qids for single operation ========================================================== 07:06:38 (1787396798)
[11733.965977] Lustre: DEBUG MARKER: == sanity-quota test 83: Setting default quota shouldn't affect grace time ========================================================== 07:07:11 (1787396831)
[11767.125773] Lustre: DEBUG MARKER: == sanity-quota test 84: Reset quota should fix the insane granted quota ========================================================== 07:07:44 (1787396864)
[11820.647509] Lustre: *** cfs_fail_loc=a08, val=0***
[11820.649895] Lustre: Skipped 29 previous similar messages
[11820.661227] Lustre: *** cfs_fail_loc=a08, val=0***
[11820.665106] Lustre: Skipped 12 previous similar messages
[11934.971656] Lustre: DEBUG MARKER: == sanity-quota test 85: do not hung at write with the least_qunit ========================================================== 07:10:32 (1787397032)
[12021.853174] Lustre: DEBUG MARKER: == sanity-quota test 86: Pre-acquired quota should be released if quota is over limit ========================================================== 07:11:59 (1787397119)
[12101.465221] LustreError: 239602:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1788002000 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[12110.659462] Lustre: DEBUG MARKER: sanity-quota test_86: @@@@@@ FAIL: succeed to create files, expect failed