[ 5234.623645] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5241.347850] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 5245.173479] LustreError: 121723:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5246.081463] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5247.616980] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5250.156432] Lustre: Failing over lustre-MDT0000
[ 5250.336700] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.34@tcp (stopping)
[ 5250.656853] Lustre: server umount lustre-MDT0000 complete
[ 5269.472793] Lustre: 3280:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787795567/real 1787795567]  req@ffff93b1497b5a40 x1874634073244160/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1787795583 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 5269.473872] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5269.475303] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5269.507950] Lustre: 3280:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
[ 5269.542282] Lustre: Skipped 1 previous similar message
[ 5273.888219] Lustre: 3282:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787795572/real 1787795572]  req@ffff93b1497b5680 x1874634073244800/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787795588 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5279.520201] Lustre: 3281:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787795577/real 1787795577]  req@ffff93b17ff992c0 x1874634073245312/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787795593 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5279.561189] Lustre: 3281:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5279.714960] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x53baad736a028f70
[ 5279.721023] Lustre: MGC192.168.201.134@tcp: Connection restored to 0@lo (at 0@lo)
[ 5280.285104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5280.356752] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5285.173708] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5290.215673] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5290.409982] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5290.507168] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:676 to 0x280000400:705)
[ 5290.509170] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:679 to 0x240000400:705)
[ 5296.568499] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5298.547836] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5314.373277] Lustre: DEBUG MARKER: (dd_pid=115072, time=9, timeout=600)
[ 5356.767698] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5363.279430] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 5366.318039] LustreError: 124580:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5367.183854] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5368.797755] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5371.099723] Lustre: Failing over lustre-MDT0000
[ 5371.369679] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5371.390091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5371.502316] Lustre: server umount lustre-MDT0000 complete
[ 5388.855415] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5389.465890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5389.579280] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5391.459912] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 5391.476721] Lustre: Skipped 2 previous similar messages
[ 5393.755282] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5394.285403] LustreError: 125227:0:(tgt_handler.c:534:tgt_filter_recovery_request()) @@@ not permitted during recovery  req@ffff93b0445970c0 x1874634073297664/t0(0) o601->lustre-MDT0000-lwp-OST0000_UUID@0@lo:264/0 lens 336/0 e 0 to 0 dl 1787795719 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'qsd_reint_0.lus.0' uid:0 gid:0 projid:4294967295
[ 5394.313369] LustreError: lustre-MDT0000-lwp-OST0000: operation quota_acquire to node 0@lo failed: rc = -11
[ 5398.248074] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5398.461464] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5398.516890] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:707 to 0x240000400:737)
[ 5398.517627] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:676 to 0x280000400:737)
[ 5404.214443] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5405.715551] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5410.390322] Lustre: DEBUG MARKER: (dd_pid=117446, time=0, timeout=600)
[ 5472.185443] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 21:56:24 (1787795784)
[ 5493.092633] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 5494.877696] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 5500.467773] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 5502.338582] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 5538.600586] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 21:57:31 (1787795851)
[ 5562.180550] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 5578.613000] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 5580.377557] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 5582.072575] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 5583.964933] Lustre: DEBUG MARKER: Set quota for 1 times
[ 5587.522294] Lustre: DEBUG MARKER: Set quota for 2 times
[ 5591.253218] Lustre: DEBUG MARKER: Set quota for 3 times
[ 5595.248194] Lustre: DEBUG MARKER: Set quota for 4 times
[ 5598.846696] Lustre: DEBUG MARKER: Set quota for 5 times
[ 5602.664681] Lustre: DEBUG MARKER: Set quota for 6 times
[ 5606.112671] Lustre: DEBUG MARKER: Set quota for 7 times
[ 5609.853625] Lustre: DEBUG MARKER: Set quota for 8 times
[ 5613.532979] Lustre: DEBUG MARKER: Set quota for 9 times
[ 5670.718548] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 21:59:43 (1787795983)
[ 5683.680915] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5683.701730] Lustre: Skipped 1 previous similar message
[ 5683.707582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5683.718665] Lustre: Skipped 2 previous similar messages
[ 5684.532754] Lustre: server umount lustre-MDT0000 complete
[ 5689.234486] LustreError: 5805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787796003 with bad export cookie 6033325362027599808
[ 5689.242575] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5689.338133] Lustre: server umount lustre-OST0000 complete
[ 5693.680978] Lustre: server umount lustre-OST0001 complete
[ 5708.802199] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local
[ 5719.041953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5724.260407] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5727.491612] Lustre: 134453:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 5734.428963] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 5742.005418] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5744.679178] LustreError: 134812:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5744.712047] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:740 to 0x240000400:769)
[ 5749.729273] LustreError: 134812:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5751.440207] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 5756.407095] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:739 to 0x280000400:769)
[ 5758.625489] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5766.877220] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 5770.912557] Lustre: 136377:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 5787.105045] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5787.139706] Lustre: Skipped 1 previous similar message
[ 5787.149631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5788.438123] Lustre: server umount lustre-MDT0000 complete
[ 5792.957769] LustreError: 134002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787796107 with bad export cookie 6033325362027607284
[ 5792.972518] LustreError: 134002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 5792.973482] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5793.037936] Lustre: server umount lustre-OST0000 complete
[ 5797.792702] Lustre: server umount lustre-OST0001 complete
[ 5813.996686] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local
[ 5827.176707] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5833.814425] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5836.471656] Lustre: 139042:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 5850.208580] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5852.131157] LustreError: 139411:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5852.199152] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:740 to 0x240000400:801)
[ 5857.265083] LustreError: 139411:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5860.498879] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 5860.508577] Lustre: Skipped 1 previous similar message
[ 5865.465647] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:739 to 0x280000400:801)
[ 5866.790255] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5874.815897] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 5878.655200] Lustre: 140960:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 5888.772029] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 22:03:21 (1787796201)
[ 5890.355980] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 5892.023418] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 22:03:24 (1787796204)
[ 5952.303625] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 22:04:24 (1787796264)
[ 5999.583947] Lustre: DEBUG MARKER: Write...
[ 6002.181630] Lustre: DEBUG MARKER: Write out of block quota ...
[ 6056.103672] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 22:06:08 (1787796368)
[ 6062.528987] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 22:06:15 (1787796375)
[ 6071.983064] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 22:06:24 (1787796384)
[ 6081.720579] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 22:06:34 (1787796394)
[ 6089.960485] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 22:06:42 (1787796402)
[ 6153.772616] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 6310.750253] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 6493.581382] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 22:13:26 (1787796806)
[ 6553.999116] Lustre: DEBUG MARKER: Restart...
[ 6559.448263] Lustre: server umount lustre-MDT0000 complete
[ 6563.137289] LustreError: 138592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787796877 with bad export cookie 6033325362027608817
[ 6563.143938] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6573.653635] Lustre: server umount lustre-OST0000 complete
[ 6586.024265] Lustre: server umount lustre-OST0001 complete
[ 6599.658620] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local
[ 6609.746992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6614.293533] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6618.036371] Lustre: 159423:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6624.804883] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 6631.362057] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6634.039133] LustreError: 159797:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6634.095868] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:811 to 0x240000400:833)
[ 6639.073772] LustreError: 159797:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6640.277973] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6645.234639] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:809 to 0x280000400:833)
[ 6648.177678] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6655.377714] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6658.473362] Lustre: 161348:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6719.822414] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 22:17:12 (1787797032)
[ 6793.833218] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 22:18:26 (1787797106)
[ 8195.926974] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 22:41:48 (1787798508)
[ 8209.348742] Lustre: server umount lustre-MDT0000 complete
[ 8213.083322] LustreError: 158977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787798527 with bad export cookie 6033325362027615313
[ 8213.090782] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 8223.888755] Lustre: server umount lustre-OST0000 complete
[ 8227.809259] Lustre: 3281:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787798526/real 1787798526]  req@ffff93b052b292c0 x1874634076589184/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787798542 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 8227.835159] Lustre: 3281:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 8227.839943] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 8227.847947] Lustre: Skipped 1 previous similar message
[ 8228.081830] Lustre: server umount lustre-OST0001 complete
[ 8246.752497] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local
[ 8258.778500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 8264.147485] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8266.912926] Lustre: 169616:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 8274.074084] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 8281.244381] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8283.243679] LustreError: 169991:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 8283.272087] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5835 to 0x240000400:5857)
[ 8288.739530] LustreError: 169991:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 8289.014172] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 8294.412733] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5835 to 0x280000400:5857)
[ 8295.117808] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8301.117133] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 8304.241942] Lustre: 171544:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 8341.465116] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 22:44:14 (1787798654)
[ 8387.063475] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 22:44:59 (1787798699)
[ 8433.778940] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 22:45:46 (1787798746)
[ 8435.206142] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs
[ 8436.481907] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 22:45:49 (1787798749)
[ 8437.966350] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs
[ 8439.475886] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 22:45:52 (1787798752)
[ 8518.626648] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 22:47:11 (1787798831)
[ 8531.575262] LustreError: 3282:0:(qsd_reint.c:633:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:0 until pending updates are flushed.
[ 8566.902448] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 22:47:59 (1787798879)
[ 8649.461316] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 22:49:22 (1787798962)
[ 8660.213852] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8660.221699] Lustre: Skipped 2 previous similar messages
[ 8662.227043] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8662.228908] Lustre: Skipped 133 previous similar messages
[ 8666.233985] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8666.239585] Lustre: Skipped 253 previous similar messages
[ 8674.233909] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8674.238844] Lustre: Skipped 476 previous similar messages
[ 8690.233695] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8690.236450] Lustre: Skipped 1023 previous similar messages
[ 8722.251929] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8722.254016] Lustre: Skipped 1878 previous similar messages
[ 8786.271400] Lustre: *** cfs_fail_loc=a09, val=0***
[ 8786.273246] Lustre: Skipped 4007 previous similar messages
[ 9068.068518] Lustre: *** cfs_fail_loc=a08, val=0***
[ 9068.080691] Lustre: Skipped 228 previous similar messages
[ 9068.094098] Lustre: *** cfs_fail_loc=a08, val=0***
[ 9068.104545] Lustre: Skipped 4 previous similar messages
[ 9199.204326] Lustre: *** cfs_fail_loc=a08, val=0***
[ 9199.206187] Lustre: Skipped 12 previous similar messages
[ 9308.133890] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 9308.143404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 9308.146505] Lustre: Skipped 1 previous similar message
[ 9313.249263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 9313.254509] Lustre: Skipped 2 previous similar messages
[ 9314.587568] Lustre: server umount lustre-MDT0000 complete
[ 9318.398276] LustreError: 175317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787799632 with bad export cookie 6033325362029387370
[ 9318.426178] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 9318.844314] Lustre: server umount lustre-OST0000 complete
[ 9323.016590] Lustre: server umount lustre-OST0001 complete
[ 9329.197401] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid
[ 9337.292590] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local
[ 9348.574404]  vdc: vdc1 vdc9
[ 9357.708102]  vde: vde1 vde9
[ 9367.490310]  vdf: vdf1 vdf9
[ 9383.951523] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local
[ 9392.496235] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000
[ 9392.666425] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[ 9392.728680] Lustre: lustre-MDT0000: new disk, initializing
[ 9393.106143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 9393.211693] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[ 9398.249629] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9403.459639] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params
[ 9409.205093] Lustre: lustre-OST0000: new disk, initializing
[ 9409.209377] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[ 9409.231336] Lustre: Skipped 1 previous similar message
[ 9409.391402] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 9410.821438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost
[ 9410.825782] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost]
[ 9410.942868] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400
[ 9415.679293] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9425.253765] Lustre: lustre-OST0001: new disk, initializing
[ 9425.261636] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[ 9425.351822] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 9427.351039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost
[ 9427.365529] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost]
[ 9427.531496] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400
[ 9432.202730] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9441.674799] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 9444.649902] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params
[ 9485.831422] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 23:03:17 (1787799797)
[ 9494.143945] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 23:03:26 (1787799806)
[ 9543.470758] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 23:04:15 (1787799855)
[ 9609.076741] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 23:05:21 (1787799921)
[ 9641.138281] Lustre: DEBUG MARKER: rename directory return 255
[ 9689.139673] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 23:06:41 (1787800001)
[ 9719.788977] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 23:07:12 (1787800032)
[ 9760.192911] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 23:07:52 (1787800072)
[ 9900.065341] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 23:10:12 (1787800212)
[ 9940.180166] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 23:10:52 (1787800252)
[ 9988.174647] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 23:11:40 (1787800300)
[10012.396070] LustreError: 3280:0:(qsd_reint.c:633:qqi_reint_delayed()) lustre-OST0000: Delaying reintegration for qtype:2 until pending updates are flushed.
[10158.489538] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 23:14:30 (1787800470)
[10160.178692] Lustre: DEBUG MARKER: SKIP: sanity-quota test_59 ldiskfs only test
[10162.317177] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 23:14:34 (1787800474)
[10223.509803] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[10225.350896] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 23:15:37 (1787800537)
[10257.995968] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[10259.968745] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 23:16:12 (1787800572)
[10313.621577] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[10315.475637] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 23:17:07 (1787800627)
[10365.484625] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 23:17:57 (1787800677)
[10367.249246] Lustre: DEBUG MARKER: SKIP: sanity-quota test_67 ZFS grants some block space together with inode
[10369.071225] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 23:18:01 (1787800681)
[10397.466611] LustreError: 209772:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[10442.729799] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 23:19:15 (1787800755)
[10471.520863] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[10473.287878] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[10651.474464] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 23:22:43 (1787800963)
[10706.119926] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 23:23:38 (1787801018)
[10741.013413] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 23:24:13 (1787801053)
[10742.515214] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71a ZFS grants some block space together with inode
[10744.387598] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 23:24:16 (1787801056)
[10745.821790] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71b ZFS grants some block space together with inode
[10747.248853] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 23:24:20 (1787801060)
[10765.619083] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB)
[10782.418452] Lustre: DEBUG MARKER: Write...
[10785.317704] Lustre: DEBUG MARKER: Write out of block quota ...
[10848.977141] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 23:26:01 (1787801161)
[10876.785041] Lustre: DEBUG MARKER: set to use default quota
[10878.557565] Lustre: DEBUG MARKER: set default quota
[10880.150915] Lustre: DEBUG MARKER: get default quota
[10885.468210] Lustre: DEBUG MARKER: Test not out of quota
[10890.222850] Lustre: DEBUG MARKER: Test out of quota
[10903.179790] Lustre: DEBUG MARKER: Increase default quota
[10934.954442] Lustre: DEBUG MARKER: Set quota to override default quota
[10935.053211] LustreError: 186539:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45063 time:1788406049 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[10947.902381] Lustre: DEBUG MARKER: Set to use default quota again
[10967.112705] Lustre: DEBUG MARKER: Cleanup
[11041.538514] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 23:29:13 (1787801353)
[11063.613388] Lustre: DEBUG MARKER: set default quota for qpool1
[11065.269950] Lustre: DEBUG MARKER: Write from user that hasn't lqe
[11117.547564] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 23:30:30 (1787801430)
[11204.146097] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 23:31:56 (1787801516)
[11287.162568] Lustre: DEBUG MARKER: Write...
[11291.378885] Lustre: DEBUG MARKER: Write out of block quota ...
[11403.653462] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 23:35:16 (1787801716)
[11452.159932] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 23:36:04 (1787801764)
[11459.755641] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 23:36:12 (1787801772)
[11462.022912] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78A need >= 2.13.57 and ldiskfs for fallocate
[11463.821431] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 23:36:16 (1787801776)
[11465.316375] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78a need >= 2.13.57 and ldiskfs for fallocate
[11467.028532] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 23:36:19 (1787801779)
[11486.766828] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 23:36:39 (1787801799)
[11488.075992] Lustre: DEBUG MARKER: SKIP: sanity-quota test_80 ZFS grants some block space together with inode
[11489.774985] Lustre: DEBUG MARKER: == sanity-quota test 81: Race qmt_start_pool_recalc with qmt_pool_free ========================================================== 23:36:42 (1787801802)
[11505.742114] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[11511.853718] LustreError: 235109:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 sleeping for 10000ms
[11514.621610] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.34@tcp (stopping)
[11516.384798] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[11516.402630] Lustre: Skipped 2 previous similar messages
[11516.405885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[11516.426855] Lustre: Skipped 1 previous similar message
[11519.733372] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.34@tcp (stopping)
[11521.854166] LustreError: 235109:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 awake
[11522.140718] Lustre: server umount lustre-MDT0000 complete
[11530.969331] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[11531.308711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[11531.420638] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:27 to 0x240000400:65)
[11531.422812] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:27 to 0x280000400:65)
[11536.497582] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[11546.622948] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[11546.633338] Lustre: Skipped 1 previous similar message
[11575.761033] Lustre: DEBUG MARKER: == sanity-quota test 82: verify more than 8 qids for single operation ========================================================== 23:38:08 (1787801888)
[11611.908565] Lustre: DEBUG MARKER: == sanity-quota test 83: Setting default quota shouldn't affect grace time ========================================================== 23:38:44 (1787801924)
[11646.651386] Lustre: DEBUG MARKER: == sanity-quota test 84: Reset quota should fix the insane granted quota ========================================================== 23:39:19 (1787801959)
[11712.697973] Lustre: *** cfs_fail_loc=a08, val=0***
[11712.705518] Lustre: Skipped 24 previous similar messages
[11712.719197] Lustre: *** cfs_fail_loc=a08, val=0***
[11712.722993] Lustre: Skipped 14 previous similar messages
[11826.447539] Lustre: DEBUG MARKER: == sanity-quota test 85: do not hung at write with the least_qunit ========================================================== 23:42:19 (1787802139)
[11918.876883] Lustre: DEBUG MARKER: == sanity-quota test 86: Pre-acquired quota should be released if quota is over limit ========================================================== 23:43:51 (1787802231)
[11995.020901] LustreError: 235694:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1788407109 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[12168.389366] LustreError: 235693:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1788407282 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[12177.333207] Lustre: DEBUG MARKER: sanity-quota test_86: @@@@@@ FAIL: succeed to create files, expect failed