[ 3158.560205] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 3159.879615] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 3160.973448] LustreError: 9615:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 3161.309560] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3161.815116] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 3162.378256] Lustre: Failing over lustre-MDT0000
[ 3162.493024] LustreError: 9993:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup
[ 3162.494971] LustreError: 9993:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message
[ 3162.560886] Lustre: server umount lustre-MDT0000 complete
[ 3175.178530] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3175.342811] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 3175.351178] Lustre: Skipped 1 previous similar message
[ 3175.459294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3175.510147] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 3176.943024] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3178.920081] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 3179.012181] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 3179.035534] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:685 to 0x280000400:705)
[ 3179.035903] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:689 to 0x240000400:705)
[ 3179.344445] Lustre: 3294:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746148511/real 1746148511]  req@ffff88008c7cb480 x1830966138570752/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1746148527 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0
[ 3179.356426] Lustre: 3294:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 3180.330144] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3180.365458] Lustre: 3294:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746148512/real 1746148512]  req@ffff88009bc7fb80 x1830966138571136/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1746148528 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0
[ 3180.411124] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to  (at 0@lo)
[ 3180.955915] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3181.397764] LustreError: 10195:0:(qsd_reint.c:617:qqi_reint_delayed()) lustre-OST0000: Delaying reintegration for qtype:0 until pending updates are flushed.
[ 3181.409291] LustreError: 10195:0:(qsd_reint.c:617:qqi_reint_delayed()) Skipped 6 previous similar messages
[ 3185.385534] Lustre: 3292:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746148517/real 1746148517]  req@ffff88009bc7f100 x1830966138571520/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1746148533 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0
[ 3185.402989] Lustre: 3292:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 3187.995072] Lustre: DEBUG MARKER: (dd_pid=2240, time=4, timeout=600)
[ 3190.414570] Lustre: 3291:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746148522/real 1746148522]  req@ffff88008ea48a80 x1830966138571904/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1746148538 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0
[ 3190.434540] Lustre: 3291:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 3218.655764] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 3220.537962] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 3221.877403] LustreError: 14039:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 3222.296493] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3223.269686] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 3224.278934] Lustre: Failing over lustre-MDT0000
[ 3224.409501] LustreError: 14861:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup
[ 3224.413630] LustreError: 14861:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 3224.501080] Lustre: server umount lustre-MDT0000 complete
[ 3237.568893] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3237.864343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3237.924890] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 3239.030200] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 3239.104419] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 3239.130417] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:707 to 0x240000400:737)
[ 3239.130600] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:685 to 0x280000400:737)
[ 3239.555858] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3239.759773] Lustre: 3291:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746148572/real 1746148572]  req@ffff88012dd2ca80 x1830966138598784/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1746148588 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0
[ 3239.768521] Lustre: 3291:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 3243.295022] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3244.002001] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3250.831602] Lustre: 3292:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746148583/real 1746148583]  req@ffff8801355a8700 x1830966138599808/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1746148599 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0
[ 3250.850278] Lustre: 3292:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
[ 3254.177642] Lustre: DEBUG MARKER: (dd_pid=4601, time=7, timeout=600)
[ 3292.083832] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 21:17:20 (1746148640)
[ 3300.297409] LustreError: 19802:0:(qsd_reint.c:617:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:1 until pending updates are flushed.
[ 3300.306886] LustreError: 19802:0:(qsd_reint.c:617:qqi_reint_delayed()) Skipped 1 previous similar message
[ 3303.871006] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 3304.652340] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 3306.551841] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 3307.225281] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 3327.534824] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 21:17:55 (1746148675)
[ 3336.453329] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 3345.378642] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 3346.068687] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 3346.724916] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 3347.509712] Lustre: DEBUG MARKER: Set quota for 1 times
[ 3349.284259] Lustre: DEBUG MARKER: Set quota for 2 times
[ 3351.073535] Lustre: DEBUG MARKER: Set quota for 3 times
[ 3352.854104] Lustre: DEBUG MARKER: Set quota for 4 times
[ 3354.501599] Lustre: DEBUG MARKER: Set quota for 5 times
[ 3356.235933] Lustre: DEBUG MARKER: Set quota for 6 times
[ 3358.224283] Lustre: DEBUG MARKER: Set quota for 7 times
[ 3360.052998] Lustre: DEBUG MARKER: Set quota for 8 times
[ 3361.979123] Lustre: DEBUG MARKER: Set quota for 9 times
[ 3363.974453] Lustre: DEBUG MARKER: Set quota for 10 times
[ 3365.841163] Lustre: DEBUG MARKER: Set quota for 11 times
[ 3367.686925] Lustre: DEBUG MARKER: Set quota for 12 times
[ 3369.547677] Lustre: DEBUG MARKER: Set quota for 13 times
[ 3371.491956] Lustre: DEBUG MARKER: Set quota for 14 times
[ 3373.814053] Lustre: DEBUG MARKER: Set quota for 15 times
[ 3376.181528] Lustre: DEBUG MARKER: Set quota for 16 times
[ 3408.219106] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 21:19:16 (1746148756)
[ 3416.400195] LustreError: 28598:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup
[ 3416.405562] LustreError: 28598:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 3416.475344] Lustre: server umount lustre-MDT0000 complete
[ 3418.567780] LustreError: 1223:0:(ldlm_lockd.c:2549:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1746148766 with bad export cookie 14151237834047469547
[ 3418.575182] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3428.624301] Lustre: server umount lustre-OST0000 complete
[ 3440.499201] Lustre: server umount lustre-OST0001 complete
[ 3446.143474] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local
[ 3450.008506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3451.513096] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3453.088118] Lustre: Modifying parameter general.debug_raw_pointers in log params
[ 3456.176971] Lustre: 31578:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.204.133@tcp: no remote llog for lustre-sptlrpc, check MGS config
[ 3456.370615] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 3459.104827] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3460.347414] LustreError: 31581:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3460.356594] LustreError: 31581:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message
[ 3460.376032] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:739 to 0x240000400:769)
[ 3462.780083] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 3464.140629] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:740 to 0x280000400:769)
[ 3465.153011] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3469.550315] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 3471.499331] Lustre: Modifying parameter general.lod.*.mdt_hash in log params
[ 3486.462830] LustreError: 2332:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup
[ 3486.467150] LustreError: 2332:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages
[ 3486.590456] Lustre: server umount lustre-MDT0000 complete
[ 3488.689053] LustreError: 30561:0:(ldlm_lockd.c:2549:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1746148836 with bad export cookie 14151237834047483379
[ 3488.695585] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3498.742569] Lustre: server umount lustre-OST0000 complete
[ 3510.806932] Lustre: server umount lustre-OST0001 complete
[ 3515.517045] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local
[ 3518.903940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3520.354947] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3521.801822] Lustre: Modifying parameter general.debug_raw_pointers in log params
[ 3524.428472] Lustre: 5311:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.204.133@tcp: no remote llog for lustre-sptlrpc, check MGS config
[ 3524.432630] Lustre: 5311:0:(mgc_request_server.c:550:mgc_llog_local_copy()) Skipped 1 previous similar message
[ 3527.377979] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3529.588316] LustreError: 5315:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3529.606370] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:739 to 0x240000400:801)
[ 3532.469334] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:740 to 0x280000400:801)
[ 3533.883239] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3538.547165] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 3540.724111] Lustre: Modifying parameter general.lod.*.mdt_hash in log params
[ 3550.313296] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 21:21:38 (1746148898)
[ 3551.011294] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 3551.759404] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 21:21:39 (1746148899)
[ 3584.119145] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 21:22:12 (1746148932)
[ 3611.463104] Lustre: DEBUG MARKER: Write...
[ 3612.564338] Lustre: DEBUG MARKER: Write out of block quota ...
[ 3650.754358] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 21:23:18 (1746148998)
[ 3653.374899] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 21:23:21 (1746149001)
[ 3658.950392] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 21:23:26 (1746149006)
[ 3663.049169] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 21:23:31 (1746149011)
[ 3666.363354] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 21:23:34 (1746149014)
[ 3708.570448] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 3784.910682] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 3902.096456] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 21:27:30 (1746149250)
[ 3935.220776] Lustre: DEBUG MARKER: Restart...
[ 3936.776388] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 3936.776673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 3936.792054] Lustre: Skipped 3 previous similar messages
[ 3941.783201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 3941.785739] Lustre: Skipped 2 previous similar messages
[ 3942.727508] LustreError: 31212:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup
[ 3942.730568] LustreError: 31212:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages
[ 3942.803405] Lustre: server umount lustre-MDT0000 complete
[ 3944.267408] LustreError: 17866:0:(ldlm_lockd.c:2549:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1746149292 with bad export cookie 14151237834047484912
[ 3944.270382] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3944.278841] LustreError: 17866:0:(ldlm_lockd.c:2549:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 3944.310775] Lustre: server umount lustre-OST0000 complete
[ 3945.789458] Lustre: server umount lustre-OST0001 complete
[ 3950.800017] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local
[ 3953.654126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3953.657229] Lustre: Skipped 2 previous similar messages
[ 3954.957820] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3956.211158] Lustre: Modifying parameter general.debug_raw_pointers in log params
[ 3958.391207] Lustre: 1711:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.204.133@tcp: no remote llog for lustre-sptlrpc, check MGS config
[ 3958.397187] Lustre: 1711:0:(mgc_request_server.c:550:mgc_llog_local_copy()) Skipped 1 previous similar message
[ 3960.476204] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3964.212401] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:811 to 0x240000400:833)
[ 3964.218519] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:809 to 0x280000400:833)
[ 3964.977841] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3968.446272] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 3969.950384] Lustre: Modifying parameter general.lod.*.mdt_hash in log params
[ 4003.589960] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 21:29:11 (1746149351)
[ 4053.043592] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 21:30:01 (1746149401)
[ 4918.501018] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 21:44:26 (1746150266)
[ 4928.397335] LustreError: 24707:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup
[ 4928.400252] LustreError: 24707:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages
[ 4928.482087] Lustre: server umount lustre-MDT0000 complete
[ 4930.292798] LustreError: 692:0:(ldlm_lockd.c:2549:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1746150278 with bad export cookie 14151237834047491296
[ 4930.298569] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4940.341608] Lustre: server umount lustre-OST0000 complete
[ 4952.114104] Lustre: server umount lustre-OST0001 complete
[ 4957.418385] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local
[ 4961.240452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4961.243855] Lustre: Skipped 2 previous similar messages
[ 4962.547600] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4963.897028] Lustre: Modifying parameter general.debug_raw_pointers in log params
[ 4966.375881] Lustre: 27710:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.204.133@tcp: no remote llog for lustre-sptlrpc, check MGS config
[ 4966.384509] Lustre: 27710:0:(mgc_request_server.c:550:mgc_llog_local_copy()) Skipped 1 previous similar message
[ 4966.524032] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 4968.545915] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4971.495413] Lustre: 28760:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.204.133@tcp: no remote llog for lustre-sptlrpc, check MGS config
[ 4971.505861] LustreError: 28763:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 4971.518060] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5836 to 0x240000400:5857)
[ 4973.555390] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4973.631803] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5834 to 0x280000400:5857)
[ 4976.999087] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 4978.284965] Lustre: Modifying parameter general.lod.*.mdt_hash in log params
[ 5006.058447] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 21:45:53 (1746150353)
[ 5032.684217] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 21:46:20 (1746150380)
[ 5060.419683] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 21:46:48 (1746150408)
[ 5060.901458] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs
[ 5061.469292] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 21:46:49 (1746150409)
[ 5061.942840] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs
[ 5062.548510] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 21:46:50 (1746150410)
[ 5071.255603] LustreError: 1475:0:(qsd_reint.c:617:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:0 until pending updates are flushed.
[ 5071.264443] LustreError: 1475:0:(qsd_reint.c:617:qqi_reint_delayed()) Skipped 3 previous similar messages
[ 5099.775804] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 21:47:27 (1746150447)
[ 5126.519726] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 21:47:54 (1746150474)
[ 5191.555861] Lustre: DEBUG MARKER: == sanity-quota test 49: lfs quota -a prints the quota usage for all quota IDs ========================================================== 21:48:59 (1746150539)
[ 5198.608620] Lustre: *** cfs_fail_loc=a09, val=0***
[ 5198.611439] Lustre: Skipped 1 previous similar message
[ 5200.615067] Lustre: *** cfs_fail_loc=a09, val=0***
[ 5200.616675] Lustre: Skipped 261 previous similar messages
[ 5204.621555] Lustre: *** cfs_fail_loc=a09, val=0***
[ 5204.623922] Lustre: Skipped 455 previous similar messages
[ 5212.625603] Lustre: *** cfs_fail_loc=a09, val=0***
[ 5212.627229] Lustre: Skipped 847 previous similar messages
[ 5228.632754] Lustre: *** cfs_fail_loc=a09, val=0***
[ 5228.634622] Lustre: Skipped 1745 previous similar messages
[ 5260.649405] Lustre: *** cfs_fail_loc=a09, val=0***
[ 5260.652881] Lustre: Skipped 3165 previous similar messages
[ 5567.640405] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5567.651576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5567.654231] Lustre: Skipped 1 previous similar message
[ 5571.052389] LustreError: 22555:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup
[ 5571.055515] LustreError: 22555:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages
[ 5571.165622] Lustre: server umount lustre-MDT0000 complete
[ 5573.624789] LustreError: 26677:0:(ldlm_lockd.c:2549:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1746150921 with bad export cookie 14151237834049253497
[ 5573.638752] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5573.867707] Lustre: server umount lustre-OST0000 complete
[ 5575.849277] LustreError: 23241:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup
[ 5575.852237] LustreError: 23241:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages
[ 5575.939231] Lustre: server umount lustre-OST0001 complete
[ 5578.808187] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid
[ 5581.758132] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local
[ 5586.514447]  vdc: vdc1 vdc9
[ 5590.569161]  vde: vde1 vde9
[ 5594.500202]  vdf: vdf1 vdf9
[ 5600.370286] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local
[ 5603.451956] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000
[ 5603.532802] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[ 5603.585155] Lustre: lustre-MDT0000: new disk, initializing
[ 5603.733587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5603.737126] Lustre: Skipped 1 previous similar message
[ 5603.765975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[ 5605.012930] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5607.576644] Lustre: Modifying parameter general.debug_raw_pointers in log params
[ 5610.189288] Lustre: 28866:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.204.133@tcp: no remote llog for lustre-sptlrpc, check MGS config
[ 5610.207206] Lustre: lustre-OST0000: new disk, initializing
[ 5610.210811] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[ 5610.213157] Lustre: Skipped 1 previous similar message
[ 5610.259416] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 5612.230808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost
[ 5612.241649] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost]
[ 5612.341193] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400
[ 5612.570635] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5617.130259] Lustre: 30181:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.204.133@tcp: no remote llog for lustre-sptlrpc, check MGS config
[ 5617.143180] Lustre: lustre-OST0001: new disk, initializing
[ 5617.146445] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[ 5617.181976] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 5618.358688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost
[ 5618.365022] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost]
[ 5618.450848] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400
[ 5619.512650] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5625.098649] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 5626.995366] Lustre: Setting parameter general.lod.*.mdt_hash in log params
[ 5653.343397] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 21:56:41 (1746151001)
[ 5687.330671] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 21:57:15 (1746151035)
[ 5722.383935] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 21:57:50 (1746151070)
[ 5737.504475] Lustre: DEBUG MARKER: rename directory return 255
[ 5761.925237] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 21:58:29 (1746151109)
[ 5780.719938] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 21:58:48 (1746151128)
[ 5809.894041] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 21:59:17 (1746151157)
[ 5873.069042] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 22:00:21 (1746151221)
[ 5892.071919] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 22:00:40 (1746151240)
[ 5917.980804] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 22:01:06 (1746151266)
[ 6003.577838] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 22:02:31 (1746151351)
[ 6004.282807] Lustre: DEBUG MARKER: SKIP: sanity-quota test_59 ldiskfs only test
[ 6005.115026] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 22:02:33 (1746151353)
[ 6013.926034] LustreError: 24875:0:(qsd_reint.c:617:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:0 until pending updates are flushed.
[ 6013.936389] LustreError: 24875:0:(qsd_reint.c:617:qqi_reint_delayed()) Skipped 4 previous similar messages
[ 6043.657226] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[ 6044.299395] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 22:03:12 (1746151392)
[ 6062.457262] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[ 6063.166369] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non dir/files should succeed ========================================================== 22:03:31 (1746151411)
[ 6092.457020] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[ 6093.185215] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 22:04:01 (1746151441)
[ 6123.805715] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 22:04:31 (1746151471)
[ 6124.407939] Lustre: DEBUG MARKER: SKIP: sanity-quota test_67 ZFS grants some block space together with inode
[ 6125.065987] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 22:04:33 (1746151473)
[ 6142.171277] LustreError: 31747:0:(mgs_handler.c:848:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[ 6167.692913] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 22:05:15 (1746151515)
[ 6177.854911] LustreError: 1751:0:(qsd_reint.c:617:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:2 until pending updates are flushed.
[ 6177.868090] LustreError: 1751:0:(qsd_reint.c:617:qqi_reint_delayed()) Skipped 5 previous similar messages
[ 6184.382543] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[ 6184.883960] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[ 6272.967246] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 22:07:01 (1746151621)
[ 6303.244225] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 22:07:31 (1746151651)
[ 6320.257585] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 22:07:48 (1746151668)
[ 6320.748104] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71a ZFS grants some block space together with inode
[ 6321.390127] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 22:07:49 (1746151669)
[ 6321.918767] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71b ZFS grants some block space together with inode
[ 6322.512786] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 22:07:50 (1746151670)
[ 6331.375949] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB)
[ 6337.637794] Lustre: DEBUG MARKER: Write...
[ 6338.537193] Lustre: DEBUG MARKER: Write out of block quota ...
[ 6373.821905] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 22:08:41 (1746151721)
[ 6389.640021] Lustre: DEBUG MARKER: set to use default quota
[ 6390.365525] Lustre: DEBUG MARKER: set default quota
[ 6391.044347] Lustre: DEBUG MARKER: get default quota
[ 6393.070618] Lustre: DEBUG MARKER: Test not out of quota
[ 6394.784087] Lustre: DEBUG MARKER: Test out of quota
[ 6398.390275] Lustre: DEBUG MARKER: Increase default quota
[ 6409.798760] Lustre: DEBUG MARKER: Set quota to override default quota
[ 6409.820015] LustreError: 27606:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45063 time:1746756558 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[ 6414.733875] Lustre: DEBUG MARKER: Set to use default quota again
[ 6426.915758] Lustre: DEBUG MARKER: Cleanup
[ 6476.213219] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 22:10:24 (1746151824)
[ 6491.079036] Lustre: DEBUG MARKER: set default quota for qpool1
[ 6491.891563] Lustre: DEBUG MARKER: Write from user that hasn't lqe
[ 6523.165625] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 22:11:11 (1746151871)
[ 6566.473309] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 22:11:54 (1746151914)
[ 6647.644464] Lustre: DEBUG MARKER: Write...
[ 6648.971285] Lustre: DEBUG MARKER: Write out of block quota ...
[ 6744.681029] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 22:14:52 (1746152092)
[ 6773.099694] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 22:15:21 (1746152121)
[ 6775.633969] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 22:15:23 (1746152123)
[ 6776.235352] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78A need >= 2.13.57 and ldiskfs for fallocate
[ 6776.994164] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 22:15:25 (1746152125)
[ 6777.629197] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78a need >= 2.13.57 and ldiskfs for fallocate
[ 6778.375560] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 22:15:26 (1746152126)
[ 6787.798197] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 22:15:35 (1746152135)
[ 6788.424233] Lustre: DEBUG MARKER: SKIP: sanity-quota test_80 ZFS grants some block space together with inode
[ 6789.005850] Lustre: DEBUG MARKER: == sanity-quota test 81: Race qmt_start_pool_recalc with qmt_pool_free ========================================================== 22:15:37 (1746152137)
[ 6797.514572] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[ 6802.036692] LustreError: 6748:0:(qmt_pool.c:1295:qmt_pool_recalc()) cfs_fail_timeout id a07 sleeping for 10000ms
[ 6805.255536] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6805.256085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6805.263648] Lustre: Skipped 2 previous similar messages
[ 6808.333654] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.33@tcp (stopping)
[ 6808.339715] Lustre: Skipped 1 previous similar message
[ 6810.264018] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6810.268569] Lustre: Skipped 1 previous similar message
[ 6812.067535] LustreError: 6748:0:(qmt_pool.c:1295:qmt_pool_recalc()) cfs_fail_timeout id a07 awake
[ 6812.200781] LustreError: 6882:0:(obd_class.h:479:obd_check_dev()) Device 17 not setup
[ 6812.202939] LustreError: 6882:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages
[ 6812.283675] Lustre: server umount lustre-MDT0000 complete
[ 6815.524743] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6815.759405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6815.835365] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:27 to 0x240000400:65)
[ 6815.845469] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:27 to 0x280000400:65)
[ 6817.126361] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6834.739744] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 192.168.204.133@tcp (at 0@lo)
[ 6834.742360] Lustre: Skipped 3 previous similar messages
[ 6843.053902] Lustre: DEBUG MARKER: == sanity-quota test 82: verify more than 8 qids for single operation ========================================================== 22:16:31 (1746152191)
[ 6861.362148] Lustre: DEBUG MARKER: == sanity-quota test 83: Setting default quota shouldn't affect grace time ========================================================== 22:16:49 (1746152209)
[ 6878.707827] Lustre: DEBUG MARKER: == sanity-quota test 84: Reset quota should fix the insane granted quota ========================================================== 22:17:06 (1746152226)
[ 6894.462147] LustreError: 29031:0:(qsd_reint.c:617:qqi_reint_delayed()) lustre-OST0000: Delaying reintegration for qtype:0 until pending updates are flushed.
[ 6894.467593] LustreError: 29031:0:(qsd_reint.c:617:qqi_reint_delayed()) Skipped 2 previous similar messages
[ 6904.755974] Lustre: *** cfs_fail_loc=a08, val=0***
[ 6904.757822] Lustre: Skipped 1521 previous similar messages
[ 6904.760018] Lustre: *** cfs_fail_loc=a08, val=0***
[ 6968.654154] Lustre: DEBUG MARKER: == sanity-quota test 85: do not hung at write with the least_qunit ========================================================== 22:18:36 (1746152316)
[ 7019.379275] Lustre: DEBUG MARKER: == sanity-quota test 86: Pre-acquired quota should be released if quota is over limit ========================================================== 22:19:27 (1746152367)
[ 7137.624733] LustreError: 7588:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1746757285 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[ 7306.228249] LustreError: 7588:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1746757454 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[ 7496.640631] LustreError: 7586:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:1000 enforced:1 hard:2048 soft:2048 granted:65536 time:1746757644 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[ 7565.251990] Lustre: DEBUG MARKER: == sanity-quota test 87: lfs quota -a should print default quota setting ========================================================== 22:28:33 (1746152913)
[ 7572.585134] Lustre: *** cfs_fail_loc=a09, val=0***
[ 7573.597109] Lustre: *** cfs_fail_loc=a09, val=0***
[ 7573.600766] Lustre: Skipped 105 previous similar messages
[ 7575.603271] Lustre: *** cfs_fail_loc=a09, val=0***
[ 7575.605044] Lustre: Skipped 227 previous similar messages
[ 7579.605572] Lustre: *** cfs_fail_loc=a09, val=0***
[ 7579.607634] Lustre: Skipped 548 previous similar messages
[ 7611.000833] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 7611.003904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 7611.030428] Lustre: Skipped 1 previous similar message
[ 7614.652506] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.33@tcp (stopping)
[ 7614.655364] Lustre: Skipped 1 previous similar message
[ 7616.006964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 7616.009369] Lustre: Skipped 1 previous similar message
[ 7616.863586] LustreError: 9548:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup
[ 7616.866928] LustreError: 9548:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 7616.951803] Lustre: server umount lustre-MDT0000 complete
[ 7620.833872] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 7621.086692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 7621.170530] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97)
[ 7621.171922] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:67 to 0x240000400:97)
[ 7622.731887] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7626.075981] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to  (at 0@lo)
[ 7626.079083] Lustre: Skipped 1 previous similar message
[ 7644.197569] Lustre: DEBUG MARKER: == sanity-quota test 88: Writing over quota should not hang ========================================================== 22:29:52 (1746152992)
[ 7644.866746] Lustre: DEBUG MARKER: SKIP: sanity-quota test_88 require client with >4k pages
[ 7645.532880] Lustre: DEBUG MARKER: == sanity-quota test 89: Show default quota with squash_uid ========================================================== 22:29:53 (1746152993)
[ 7685.129230] Lustre: DEBUG MARKER: == sanity-quota test 90a: lfs quota should work without mount point ========================================================== 22:30:33 (1746153033)
[ 7711.785856] Lustre: DEBUG MARKER: == sanity-quota test 90b: lfs quota should work with multiple mount points ========================================================== 22:30:59 (1746153059)
[ 7720.669735] Lustre: DEBUG MARKER: sanity-quota test_90b: @@@@@@ FAIL: quota info from /mnt/lustre not ' /mnt/lustre root 11720 0 0 - 375 0 0 -