[ 6960.991558] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 6971.637923] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 6976.824251] LustreError: 118881:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 6978.917433] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 6981.423716] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 6985.049042] Lustre: Failing over lustre-MDT0000
[ 6985.474708] Lustre: server umount lustre-MDT0000 complete
[ 7005.407118] Lustre: 3302:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785400938/real 1785400938]  req@ffff94be81ad5500 x1872121336838784/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785400954 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7005.442819] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 7005.458435] Lustre: Skipped 1 previous similar message
[ 7006.495535] Lustre: 3303:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785400938/real 1785400938]  req@ffff94bfbc402300 x1872121336838912/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1785400954 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 7006.496974] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 7006.552018] Lustre: 3303:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 7006.578289] LustreError: 3301:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight  req@ffff94bfa72e8700 x1872121336840832/t0(0) o250->MGC192.168.203.148@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7007.336445] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 7007.433311] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 7010.166175] Lustre: 3305:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785400943/real 1785400943]  req@ffff94bfbc403b80 x1872121336839168/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785400959 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7010.200991] Lustre: 3305:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 7012.072892] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 7012.316961] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 7012.429082] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:485 to 0x240000400:513)
[ 7012.431160] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:480 to 0x280000400:513)
[ 7013.370845] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7015.333515] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[ 7015.351186] Lustre: Skipped 1 previous similar message
[ 7016.351304] Lustre: 3305:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785400948/real 1785400948]  req@ffff94be86261f80 x1872121336839552/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785400964 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7016.395285] Lustre: 3305:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 7020.513736] Lustre: 3304:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785400953/real 1785400953]  req@ffff94bfa72ece00 x1872121336840064/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785400969 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7020.552585] Lustre: 3304:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 7028.972055] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 7032.894912] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 7041.029676] Lustre: DEBUG MARKER: (dd_pid=113319, time=0, timeout=600)
[ 7099.100615] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 7108.508603] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 7113.580466] LustreError: 121649:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 7115.129975] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 7117.818197] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 7120.783650] Lustre: Failing over lustre-MDT0000
[ 7121.045334] LustreError: lustre-MDT0000-lwp-OST0000: operation quota_acquire to node 0@lo failed: rc = -107
[ 7121.057331] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 7121.074451] Lustre: Skipped 1 previous similar message
[ 7121.079967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 7121.462139] Lustre: server umount lustre-MDT0000 complete
[ 7139.295237] Lustre: 3304:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785401072/real 1785401072]  req@ffff94bf8f7a1880 x1872121336885888/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785401088 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7139.334437] Lustre: 3304:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 7139.342484] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 7149.535923] LustreError: 3301:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight  req@ffff94be82ac4a80 x1872121336888704/t0(0) o250->MGC192.168.203.148@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7150.462508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 7150.670257] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 7150.829412] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 7151.092253] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 7151.183991] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:480 to 0x280000400:545)
[ 7151.188952] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:515 to 0x240000400:545)
[ 7158.088923] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7158.369376] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[ 7158.388392] Lustre: Skipped 1 previous similar message
[ 7172.281807] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 7174.621426] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 7181.672104] Lustre: DEBUG MARKER: (dd_pid=115667, time=0, timeout=600)
[ 7267.862282] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 04:46:54 (1785401214)
[ 7298.085365] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 7301.078457] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 7308.404139] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 7310.428286] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 7357.773429] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 04:48:24 (1785401304)
[ 7399.309905] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 7422.418698] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 7425.357354] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 7427.362737] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 7429.846619] Lustre: DEBUG MARKER: Set quota for 1 times
[ 7434.636384] Lustre: DEBUG MARKER: Set quota for 2 times
[ 7440.431282] Lustre: DEBUG MARKER: Set quota for 3 times
[ 7445.519657] Lustre: DEBUG MARKER: Set quota for 4 times
[ 7450.984443] Lustre: DEBUG MARKER: Set quota for 5 times
[ 7458.771289] Lustre: DEBUG MARKER: Set quota for 6 times
[ 7518.341778] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 04:51:04 (1785401464)
[ 7540.579871] Lustre: server umount lustre-MDT0000 complete
[ 7546.694154] LustreError: 24529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785401495 with bad export cookie 13702827776243497979
[ 7546.722126] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 7556.255823] Lustre: server umount lustre-OST0000 complete
[ 7560.033064] Lustre: 3302:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785401492/real 1785401492]  req@ffff94bf9f6c8e00 x1872121337026688/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785401508 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7560.073376] Lustre: 3302:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
[ 7560.103373] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 7560.132246] Lustre: Skipped 1 previous similar message
[ 7563.330563] Lustre: server umount lustre-OST0001 complete
[ 7591.678572] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local
[ 7611.761782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 7624.400125] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7629.013597] Lustre: 131116:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 7645.007912] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 7657.121919] LustreError: 131433:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 7657.178179] LustreError: 131433:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages
[ 7657.203317] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:548 to 0x240000400:577)
[ 7660.699781] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7662.563384] LustreError: 131433:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 7667.688487] LustreError: 131669:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 7673.721524] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 7679.032162] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:547 to 0x280000400:577)
[ 7684.931976] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7696.408747] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 7701.745235] Lustre: 132945:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 7725.029557] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 7725.039783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 7725.053155] Lustre: Skipped 1 previous similar message
[ 7727.441620] Lustre: server umount lustre-MDT0000 complete
[ 7733.703534] LustreError: 130659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785401682 with bad export cookie 13702827776243503810
[ 7733.709335] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 7733.725375] LustreError: 130659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 7733.940275] Lustre: server umount lustre-OST0000 complete
[ 7740.013247] Lustre: server umount lustre-OST0001 complete
[ 7760.915750] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local
[ 7776.918033] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 7784.064706] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7787.688764] Lustre: 135555:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 7795.903899] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 7801.890394] LustreError: 135874:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 7801.921845] LustreError: 135874:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message
[ 7801.923503] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:548 to 0x240000400:609)
[ 7805.150691] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7820.808504] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:547 to 0x280000400:609)
[ 7824.199568] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7834.188170] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 7838.673554] Lustre: 137378:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 7851.149316] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 04:56:38 (1785401798)
[ 7852.983406] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 7855.872609] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 04:56:42 (1785401802)
[ 7936.136466] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 04:58:02 (1785401882)
[ 8014.063263] Lustre: DEBUG MARKER: Write...
[ 8018.902091] Lustre: DEBUG MARKER: Write out of block quota ...
[ 8086.558462] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 05:00:33 (1785402033)
[ 8095.343808] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 05:00:42 (1785402042)
[ 8107.853122] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 05:00:55 (1785402055)
[ 8119.291047] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 05:01:06 (1785402066)
[ 8133.729763] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 05:01:20 (1785402080)
[ 8210.981264] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 8422.176653] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 8675.828723] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 05:10:22 (1785402622)
[ 8759.289428] Lustre: DEBUG MARKER: Restart...
[ 8766.630397] Lustre: server umount lustre-MDT0000 complete
[ 8773.710135] LustreError: 142411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785402722 with bad export cookie 13702827776243505336
[ 8773.732039] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 8783.449070] Lustre: server umount lustre-OST0000 complete
[ 8784.876908] Lustre: 3302:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785402717/real 1785402717]  req@ffff94bfbb2d1c00 x1872121337345792/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785402733 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 8784.958834] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 8788.794503] Lustre: server umount lustre-OST0001 complete
[ 8810.912721] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local
[ 8823.952383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 8823.960189] Lustre: Skipped 1 previous similar message
[ 8830.127684] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8834.642737] Lustre: 155655:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 8844.637109] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 8849.698180] LustreError: 155975:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 8849.734628] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:618 to 0x240000400:641)
[ 8849.738183] LustreError: 155975:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages
[ 8855.011348] LustreError: 155975:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 8855.386842] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8860.129529] LustreError: 155976:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 8864.956127] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 8869.903496] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:618 to 0x280000400:641)
[ 8874.553708] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8885.425503] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 8889.935171] Lustre: 157476:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 8976.662274] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 05:15:24 (1785402924)
[ 9073.847917] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 05:16:59 (1785403019)
[11624.522634] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 05:59:30 (1785405570)
[11653.102964] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[11653.108161] LustreError: 155223:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[11653.127875] Lustre: Skipped 1 previous similar message
[11653.161095] LustreError: 155223:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message
[11655.305369] Lustre: server umount lustre-MDT0000 complete
[11660.855897] LustreError: 155209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785405609 with bad export cookie 13702827776243511867
[11660.867015] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[11660.881946] LustreError: 155209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[11661.174175] Lustre: server umount lustre-OST0000 complete
[11667.963825] Lustre: server umount lustre-OST0001 complete
[11688.876513] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local
[11703.261735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[11710.008315] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[11714.321262] Lustre: 165699:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[11723.384748] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[11727.398789] LustreError: 166018:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[11727.437430] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5644 to 0x240000400:5665)
[11732.132789] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[11732.452142] LustreError: 166018:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[11737.569572] LustreError: 166179:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[11742.689600] Lustre: lustre-OST0001: Not available for connect from 0@lo (not set up)
[11742.692128] Lustre: Skipped 1 previous similar message
[11743.117967] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[11748.389112] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5642 to 0x280000400:5665)
[11752.751907] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[11761.629521] Lustre: DEBUG MARKER: Using TIMEOUT=20
[11765.449331] Lustre: 167513:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[11809.856221] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 06:02:37 (1785405757)
[11869.101419] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 06:03:35 (1785405815)
[11926.661119] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 06:04:33 (1785405873)
[11929.720991] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs
[11933.412174] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 06:04:39 (1785405879)
[11936.620722] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs
[11939.318093] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 06:04:46 (1785405886)
[12053.206187] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 06:06:39 (1785405999)
[12116.412408] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 06:07:44 (1785406064)
[12251.377495] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 06:09:58 (1785406198)
[12264.680678] Lustre: *** cfs_fail_loc=a09, val=0***
[12264.682773] Lustre: Skipped 4 previous similar messages
[12266.684065] Lustre: *** cfs_fail_loc=a09, val=0***
[12266.685701] Lustre: Skipped 67 previous similar messages
[12270.740773] Lustre: *** cfs_fail_loc=a09, val=0***
[12270.742681] Lustre: Skipped 183 previous similar messages
[12278.745957] Lustre: *** cfs_fail_loc=a09, val=0***
[12278.748545] Lustre: Skipped 313 previous similar messages
[12294.762122] Lustre: *** cfs_fail_loc=a09, val=0***
[12294.765350] Lustre: Skipped 435 previous similar messages
[12326.771446] Lustre: *** cfs_fail_loc=a09, val=0***
[12326.774035] Lustre: Skipped 799 previous similar messages
[12390.805215] Lustre: *** cfs_fail_loc=a09, val=0***
[12390.813017] Lustre: Skipped 1789 previous similar messages
[12518.848668] Lustre: *** cfs_fail_loc=a09, val=0***
[12518.871794] Lustre: Skipped 4015 previous similar messages
[13159.805288] Lustre: *** cfs_fail_loc=a08, val=0***
[13159.821831] Lustre: Skipped 405 previous similar messages
[13159.828308] Lustre: *** cfs_fail_loc=a08, val=0***
[13159.847654] Lustre: Skipped 2 previous similar messages
[13162.848860] Lustre: *** cfs_fail_loc=a08, val=0***
[13162.865842] Lustre: Skipped 39 previous similar messages
[13170.069562] Lustre: *** cfs_fail_loc=a08, val=0***
[13170.082604] Lustre: Skipped 26 previous similar messages
[13172.165654] Lustre: *** cfs_fail_loc=a08, val=0***
[13172.195730] Lustre: Skipped 345 previous similar messages
[13180.399756] Lustre: *** cfs_fail_loc=a08, val=0***
[13180.401420] Lustre: Skipped 105 previous similar messages
[13190.885204] Lustre: *** cfs_fail_loc=a08, val=0***
[13190.909789] Lustre: Skipped 397 previous similar messages
[13211.294748] Lustre: *** cfs_fail_loc=a08, val=0***
[13211.304408] Lustre: Skipped 737 previous similar messages
[13243.566107] Lustre: *** cfs_fail_loc=a08, val=0***
[13243.575309] Lustre: Skipped 1687 previous similar messages
[13331.759019] Lustre: *** cfs_fail_loc=a08, val=0***
[13331.774643] Lustre: Skipped 575 previous similar messages
[13513.697242] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[13513.698682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[13513.717747] Lustre: Skipped 1 previous similar message
[13513.740869] Lustre: Skipped 1 previous similar message
[13516.063873] Lustre: server umount lustre-MDT0000 complete
[13523.465208] LustreError: 165242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785407472 with bad export cookie 13702827776245297210
[13523.477986] LustreError: 165242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message
[13523.481727] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[13523.748188] Lustre: server umount lustre-OST0000 complete
[13530.435144] Lustre: server umount lustre-OST0001 complete
[13540.725501] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_hostid
[13556.721890] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local
[13571.618867]  vdc: vdc1 vdc9
[13585.920876]  vde: vde1 vde9
[13601.054498]  vdf: vdf1 vdf9
[13623.856576] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local
[13640.155249] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000
[13641.192783] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[13641.327891] Lustre: lustre-MDT0000: new disk, initializing
[13642.875642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[13643.169687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[13651.323559] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[13658.407273] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params
[13665.138967] Lustre: lustre-OST0000: new disk, initializing
[13665.146755] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[13665.157786] Lustre: Skipped 1 previous similar message
[13665.300203] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[13667.030671] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost
[13667.051092] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost]
[13667.217961] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400
[13673.524567] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[13688.577176] Lustre: lustre-OST0001: new disk, initializing
[13688.583244] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[13688.717490] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[13690.408869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost
[13690.443465] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost]
[13690.830941] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400
[13700.022472] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[13712.168946] Lustre: DEBUG MARKER: Using TIMEOUT=20
[13717.280367] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params
[13774.824404] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 06:35:21 (1785407721)
[13784.042975] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 06:35:30 (1785407730)
[13850.254742] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 06:36:37 (1785407797)
[13929.794629] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 06:37:55 (1785407875)
[13970.493684] Lustre: DEBUG MARKER: rename directory return 255
[14040.712421] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 06:39:44 (1785407984)
[14090.671730] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 06:40:38 (1785408038)
[14155.043993] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 06:41:41 (1785408101)
[14374.746203] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 06:45:22 (1785408322)
[14432.304777] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 06:46:19 (1785408379)
[14499.851939] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 06:47:26 (1785408446)
[14800.897404] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 06:52:27 (1785408747)
[14803.747550] Lustre: DEBUG MARKER: SKIP: sanity-quota test_59 ldiskfs only test
[14806.654872] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 06:52:33 (1785408753)
[14886.773791] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[14889.131053] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 06:53:56 (1785408836)
[14945.103091] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[14948.995448] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 06:54:54 (1785408894)
[15027.257177] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[15029.912250] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 06:56:16 (1785408976)
[15085.998974] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 06:57:12 (1785409032)
[15088.865432] Lustre: DEBUG MARKER: SKIP: sanity-quota test_67 ZFS grants some block space together with inode
[15092.687478] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 06:57:18 (1785409038)
[15131.420559] LustreError: 204932:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[15195.116258] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 06:59:01 (1785409141)
[15233.098309] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[15235.333928] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[15559.321939] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 07:05:05 (1785409505)
[15631.194786] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 07:06:17 (1785409577)
[15677.601556] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 07:07:03 (1785409623)
[15680.560515] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71a ZFS grants some block space together with inode
[15684.133826] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 07:07:10 (1785409630)
[15686.994284] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71b ZFS grants some block space together with inode
[15690.276991] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 07:07:16 (1785409636)
[15710.658535] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB)
[15733.307525] Lustre: DEBUG MARKER: Write...
[15737.626795] Lustre: DEBUG MARKER: Write out of block quota ...
[15816.381665] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 07:09:23 (1785409763)
[15853.624851] Lustre: DEBUG MARKER: set to use default quota
[15856.258637] Lustre: DEBUG MARKER: set default quota
[15858.623552] Lustre: DEBUG MARKER: get default quota
[15867.711263] Lustre: DEBUG MARKER: Test not out of quota
[15873.140093] Lustre: DEBUG MARKER: Test out of quota
[15892.620811] Lustre: DEBUG MARKER: Increase default quota
[15933.556589] Lustre: DEBUG MARKER: Set quota to override default quota
[15933.675346] LustreError: 182253:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45063 time:1786014682 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[15955.102802] Lustre: DEBUG MARKER: Set to use default quota again
[15986.357810] Lustre: DEBUG MARKER: Cleanup
[16077.493712] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 07:13:44 (1785410024)
[16113.926270] Lustre: DEBUG MARKER: set default quota for qpool1
[16117.887635] Lustre: DEBUG MARKER: Write from user that hasn't lqe
[16188.537087] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 07:15:35 (1785410135)
[16308.718857] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 07:17:35 (1785410255)
[16412.987780] Lustre: DEBUG MARKER: Write...
[16420.948310] Lustre: DEBUG MARKER: Write out of block quota ...
[16583.163675] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 07:22:10 (1785410530)
[16645.570569] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 07:23:12 (1785410592)
[16657.935841] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 07:23:24 (1785410604)
[16661.007301] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78A need >= 2.13.57 and ldiskfs for fallocate
[16664.079085] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 07:23:30 (1785410610)
[16666.462266] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78a need >= 2.13.57 and ldiskfs for fallocate
[16669.582109] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 07:23:36 (1785410616)
[16695.323563] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 07:24:02 (1785410642)
[16698.010862] Lustre: DEBUG MARKER: SKIP: sanity-quota test_80 ZFS grants some block space together with inode
[16700.196939] Lustre: DEBUG MARKER: == sanity-quota test 81: Race qmt_start_pool_recalc with qmt_pool_free ========================================================== 07:24:07 (1785410647)
[16726.621493] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[16734.651120] LustreError: 229799:0:(qmt_pool.c:1413:qmt_pool_recalc()) cfs_fail_timeout id a07 sleeping for 10000ms
[16734.667614] LustreError: 229799:0:(qmt_pool.c:1413:qmt_pool_recalc()) Skipped 5 previous similar messages
[16739.096694] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.48@tcp (stopping)
[16740.833844] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[16740.839243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[16740.869869] Lustre: Skipped 1 previous similar message
[16740.901227] Lustre: Skipped 1 previous similar message
[16744.162292] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.48@tcp (stopping)
[16744.751122] LustreError: 229799:0:(qmt_pool.c:1413:qmt_pool_recalc()) cfs_fail_timeout id a07 awake
[16745.494219] Lustre: server umount lustre-MDT0000 complete
[16758.513670] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[16759.527777] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[16760.147978] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:27 to 0x240000400:65)
[16760.159873] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:27 to 0x280000400:65)
[16760.500846] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[16760.520727] Lustre: Skipped 1 previous similar message
[16768.047573] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[16811.146574] Lustre: DEBUG MARKER: == sanity-quota test 82: verify more than 8 qids for single operation ========================================================== 07:25:58 (1785410758)
[16855.010143] Lustre: DEBUG MARKER: == sanity-quota test 83: Setting default quota shouldn't affect grace time ========================================================== 07:26:42 (1785410802)
[16897.839720] Lustre: DEBUG MARKER: == sanity-quota test 84: Reset quota should fix the insane granted quota ========================================================== 07:27:23 (1785410843)
[16977.819024] Lustre: *** cfs_fail_loc=a08, val=0***
[16977.832431] Lustre: Skipped 5914 previous similar messages
[16977.841471] Lustre: *** cfs_fail_loc=a08, val=0***
[16977.848449] Lustre: Skipped 1981 previous similar messages
[17149.220800] Lustre: DEBUG MARKER: == sanity-quota test 85: do not hung at write with the least_qunit ========================================================== 07:31:36 (1785411096)
[17259.005623] Lustre: DEBUG MARKER: == sanity-quota test 86: Pre-acquired quota should be released if quota is over limit ========================================================== 07:33:24 (1785411204)
[17369.685686] LustreError: 230330:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1786016118 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[17578.228788] LustreError: 230332:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1786016327 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[17588.537367] Lustre: DEBUG MARKER: sanity-quota test_86: @@@@@@ FAIL: succeed to create files, expect failed