[ 3345.012529] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 3348.000344] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 3349.966428] LustreError: 9396:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 3350.538188] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3351.330958] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 3352.303622] Lustre: Failing over lustre-MDT0000
[ 3352.494821] LustreError: 9833:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup
[ 3352.497496] LustreError: 9833:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message
[ 3352.505245] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.50@tcp (stopping)
[ 3352.658166] Lustre: server umount lustre-MDT0000 complete
[ 3366.176316] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3366.472413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3366.516842] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 3368.014232] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3369.474699] Lustre: 3294:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746143247/real 1746143247]  req@ffff880093561880 x1830960421789696/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1746143263 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0
[ 3369.495837] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 3369.505801] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to  (at 0@lo)
[ 3371.501753] Lustre: 3293:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746143249/real 1746143249]  req@ffff880093562300 x1830960421789952/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1746143265 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0
[ 3371.559193] LustreError: 19689:0:(qsd_reint.c:617:qqi_reint_delayed()) lustre-OST0000: Delaying reintegration for qtype:0 until pending updates are flushed.
[ 3371.570218] LustreError: 19689:0:(qsd_reint.c:617:qqi_reint_delayed()) Skipped 6 previous similar messages
[ 3371.617281] LustreError: 10455:0:(tgt_handler.c:527:tgt_filter_recovery_request()) @@@ not permitted during recovery  req@ffff8800a6c6b100 x1830960421801472/t0(0) o601->lustre-MDT0000-lwp-OST0000_UUID@0@lo:416/0 lens 336/0 e 0 to 0 dl 1746143276 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'qsd_reint_0.lus.0' uid:0 gid:0
[ 3371.640861] LustreError: lustre-MDT0000-lwp-OST0000: operation quota_acquire to node 0@lo failed: rc = -11
[ 3373.490593] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 3373.566580] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 3373.592568] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:681 to 0x240000400:705)
[ 3373.592610] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:678 to 0x280000400:705)
[ 3375.534307] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3376.507053] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3376.515602] Lustre: 3294:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746143254/real 1746143254]  req@ffff8800a6c6a680 x1830960421790336/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1746143270 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0
[ 3376.528940] Lustre: 3294:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 3380.205016] Lustre: DEBUG MARKER: (dd_pid=1488, time=0, timeout=600)
[ 3381.547779] Lustre: 3294:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746143259/real 1746143259]  req@ffff880092d4ce00 x1830960421790848/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1746143275 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0
[ 3381.564067] Lustre: 3294:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 3414.307381] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 3417.125723] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 3419.087038] LustreError: 13839:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 3419.669192] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3420.770944] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 3421.845197] Lustre: Failing over lustre-MDT0000
[ 3421.872132] LustreError: lustre-MDT0000-lwp-OST0000: operation quota_acquire to node 0@lo failed: rc = -107
[ 3421.880016] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 3422.047651] LustreError: 14582:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup
[ 3422.053141] LustreError: 14582:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 3422.175264] Lustre: server umount lustre-MDT0000 complete
[ 3436.508918] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3437.010921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3437.056861] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 3437.649081] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 3437.764081] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 3437.812150] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:678 to 0x280000400:737)
[ 3437.815354] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:707 to 0x240000400:737)
[ 3439.091624] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3439.845038] Lustre: 3293:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746143318/real 1746143318]  req@ffff88008bb70000 x1830960421821312/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1746143334 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0
[ 3439.854942] Lustre: 3293:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 3442.836969] LustreError: 19689:0:(qsd_reint.c:617:qqi_reint_delayed()) lustre-OST0000: Delaying reintegration for qtype:0 until pending updates are flushed.
[ 3443.992441] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3444.813801] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3449.947640] Lustre: 3294:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746143328/real 1746143328]  req@ffff88008bb73800 x1830960421822464/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1746143344 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0
[ 3449.968297] Lustre: 3294:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 3456.743809] Lustre: DEBUG MARKER: (dd_pid=3868, time=8, timeout=600)
[ 3498.915519] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 19:49:52 (1746143392)
[ 3512.434419] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 3513.352907] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 3515.739503] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 3516.737451] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 3539.311615] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 19:50:33 (1746143433)
[ 3551.112264] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 3561.407177] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 3562.224220] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 3563.071414] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 3564.007422] Lustre: DEBUG MARKER: Set quota for 1 times
[ 3566.202217] Lustre: DEBUG MARKER: Set quota for 2 times
[ 3568.171758] Lustre: DEBUG MARKER: Set quota for 3 times
[ 3570.187514] Lustre: DEBUG MARKER: Set quota for 4 times
[ 3572.226051] Lustre: DEBUG MARKER: Set quota for 5 times
[ 3574.210775] Lustre: DEBUG MARKER: Set quota for 6 times
[ 3576.404296] Lustre: DEBUG MARKER: Set quota for 7 times
[ 3578.362914] Lustre: DEBUG MARKER: Set quota for 8 times
[ 3580.297927] Lustre: DEBUG MARKER: Set quota for 9 times
[ 3582.430657] Lustre: DEBUG MARKER: Set quota for 10 times
[ 3584.721065] Lustre: DEBUG MARKER: Set quota for 11 times
[ 3587.045754] Lustre: DEBUG MARKER: Set quota for 12 times
[ 3589.422285] Lustre: DEBUG MARKER: Set quota for 13 times
[ 3591.806513] Lustre: DEBUG MARKER: Set quota for 14 times
[ 3617.400115] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 19:51:51 (1746143511)
[ 3619.639752] LustreError: 27313:0:(qsd_reint.c:617:qqi_reint_delayed()) lustre-OST0001: Delaying reintegration for qtype:2 until pending updates are flushed.
[ 3619.645046] LustreError: 27313:0:(qsd_reint.c:617:qqi_reint_delayed()) Skipped 1 previous similar message
[ 3625.095722] LustreError: 27910:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup
[ 3625.098317] LustreError: 27910:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 3625.204731] Lustre: server umount lustre-MDT0000 complete
[ 3627.431169] LustreError: 11081:0:(ldlm_lockd.c:2549:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1746143521 with bad export cookie 18229127965545518752
[ 3627.445573] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3637.507496] Lustre: server umount lustre-OST0000 complete
[ 3649.663988] Lustre: server umount lustre-OST0001 complete
[ 3656.606759] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local
[ 3661.451687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3663.471098] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3665.292799] Lustre: Modifying parameter general.debug_raw_pointers in log params
[ 3667.991178] Lustre: 30858:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.150@tcp: no remote llog for lustre-sptlrpc, check MGS config
[ 3668.177091] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 3668.255732] mount.lustre (30858) used greatest stack depth: 9616 bytes left
[ 3671.159447] LustreError: 30862:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3671.179629] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3671.233484] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:740 to 0x240000400:769)
[ 3675.330583] Lustre: 31924:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.150@tcp: no remote llog for lustre-sptlrpc, check MGS config
[ 3675.492848] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 3677.344245] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:739 to 0x280000400:769)
[ 3678.478161] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3683.276961] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 3685.315354] Lustre: Modifying parameter general.lod.*.mdt_hash in log params
[ 3699.986750] LustreError: 1568:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup
[ 3699.989501] LustreError: 1568:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages
[ 3700.083477] Lustre: server umount lustre-MDT0000 complete
[ 3702.397274] LustreError: 477:0:(ldlm_lockd.c:2549:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1746143596 with bad export cookie 18229127965545530540
[ 3702.412218] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3712.470558] Lustre: server umount lustre-OST0000 complete
[ 3724.677383] Lustre: server umount lustre-OST0001 complete
[ 3731.186656] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local
[ 3735.621903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3737.347252] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3739.029890] Lustre: Modifying parameter general.debug_raw_pointers in log params
[ 3741.603474] Lustre: 4569:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.150@tcp: no remote llog for lustre-sptlrpc, check MGS config
[ 3741.847085] mount.lustre (4569) used greatest stack depth: 9584 bytes left
[ 3744.083279] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3744.768417] LustreError: 4573:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3744.783059] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:740 to 0x240000400:801)
[ 3747.442022] Lustre: 5631:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.150@tcp: no remote llog for lustre-sptlrpc, check MGS config
[ 3749.341293] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:739 to 0x280000400:801)
[ 3750.694072] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3754.902364] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 3756.703912] Lustre: Modifying parameter general.lod.*.mdt_hash in log params
[ 3767.248640] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 19:54:21 (1746143661)
[ 3768.039989] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 3769.049108] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 19:54:22 (1746143662)
[ 3805.309882] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 19:54:59 (1746143699)
[ 3832.597050] Lustre: DEBUG MARKER: Write...
[ 3833.958526] Lustre: DEBUG MARKER: Write out of block quota ...
[ 3872.677056] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 19:56:06 (1746143766)
[ 3875.045567] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 19:56:09 (1746143769)
[ 3880.120679] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 19:56:14 (1746143774)
[ 3884.630943] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 19:56:18 (1746143778)
[ 3888.428963] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 19:56:22 (1746143782)
[ 3931.655889] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 4006.823617] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 4120.098152] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 20:00:14 (1746144014)
[ 4156.423809] Lustre: DEBUG MARKER: Restart...
[ 4158.350458] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 4158.351189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 4158.359763] Lustre: Skipped 4 previous similar messages
[ 4160.167351] LustreError: 29574:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup
[ 4160.172489] LustreError: 29574:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages
[ 4160.278384] Lustre: server umount lustre-MDT0000 complete
[ 4163.030156] LustreError: 3559:0:(ldlm_lockd.c:2549:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1746144057 with bad export cookie 18229127965545532073
[ 4163.032012] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4163.060974] LustreError: 3559:0:(ldlm_lockd.c:2549:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 4163.116958] Lustre: server umount lustre-OST0000 complete
[ 4165.674123] Lustre: server umount lustre-OST0001 complete
[ 4172.931416] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local
[ 4178.305980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4178.308132] Lustre: Skipped 2 previous similar messages
[ 4180.109184] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4181.613140] Lustre: Modifying parameter general.debug_raw_pointers in log params
[ 4184.108146] Lustre: 32510:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.150@tcp: no remote llog for lustre-sptlrpc, check MGS config
[ 4186.927945] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4188.258277] LustreError: 32513:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 4188.264929] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:811 to 0x240000400:833)
[ 4192.086798] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:809 to 0x280000400:833)
[ 4193.099966] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4197.282193] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 4199.188103] Lustre: Modifying parameter general.lod.*.mdt_hash in log params
[ 4236.015487] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 20:02:10 (1746144130)
[ 4286.585362] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 20:03:00 (1746144180)
[ 5082.436325] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 20:16:16 (1746144976)
[ 5091.305683] LustreError: 21723:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup
[ 5091.308009] LustreError: 21723:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages
[ 5091.433263] Lustre: server umount lustre-MDT0000 complete
[ 5093.502919] LustreError: 16888:0:(ldlm_lockd.c:2549:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1746144987 with bad export cookie 18229127965545538443
[ 5093.507834] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5103.579593] Lustre: server umount lustre-OST0000 complete
[ 5115.712554] Lustre: server umount lustre-OST0001 complete
[ 5121.303411] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local
[ 5124.964341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5124.967381] Lustre: Skipped 2 previous similar messages
[ 5126.741300] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5128.085751] Lustre: Modifying parameter general.debug_raw_pointers in log params
[ 5130.642877] Lustre: 24657:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.150@tcp: no remote llog for lustre-sptlrpc, check MGS config
[ 5130.646521] Lustre: 24657:0:(mgc_request_server.c:550:mgc_llog_local_copy()) Skipped 1 previous similar message
[ 5130.817605] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 5133.031373] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5134.804726] LustreError: 24661:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5134.811339] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5835 to 0x240000400:5857)
[ 5136.443672] Lustre: 25696:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.150@tcp: no remote llog for lustre-sptlrpc, check MGS config
[ 5138.205571] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5835 to 0x280000400:5857)
[ 5139.226315] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5143.544485] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 5145.213785] Lustre: Modifying parameter general.lod.*.mdt_hash in log params
[ 5176.729661] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 20:17:50 (1746145070)
[ 5205.615832] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 20:18:19 (1746145099)
[ 5233.618287] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 20:18:47 (1746145127)
[ 5234.274215] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs
[ 5234.916457] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 20:18:49 (1746145129)
[ 5235.526764] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs
[ 5236.186502] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 20:18:50 (1746145130)
[ 5245.811930] LustreError: 30892:0:(qsd_reint.c:617:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:0 until pending updates are flushed.
[ 5245.823495] LustreError: 30892:0:(qsd_reint.c:617:qqi_reint_delayed()) Skipped 2 previous similar messages
[ 5275.257073] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 20:19:29 (1746145169)
[ 5301.791080] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 20:19:56 (1746145196)
[ 5361.766509] Lustre: DEBUG MARKER: == sanity-quota test 49: lfs quota -a prints the quota usage for all quota IDs ========================================================== 20:20:55 (1746145255)
[ 5369.242235] Lustre: *** cfs_fail_loc=a09, val=0***
[ 5369.244082] Lustre: Skipped 1 previous similar message
[ 5370.254880] Lustre: *** cfs_fail_loc=a09, val=0***
[ 5370.258665] Lustre: Skipped 103 previous similar messages
[ 5372.257359] Lustre: *** cfs_fail_loc=a09, val=0***
[ 5372.259329] Lustre: Skipped 179 previous similar messages
[ 5376.269055] Lustre: *** cfs_fail_loc=a09, val=0***
[ 5376.270733] Lustre: Skipped 441 previous similar messages
[ 5384.274590] Lustre: *** cfs_fail_loc=a09, val=0***
[ 5384.276378] Lustre: Skipped 877 previous similar messages
[ 5400.276048] Lustre: *** cfs_fail_loc=a09, val=0***
[ 5400.279899] Lustre: Skipped 1593 previous similar messages
[ 5432.276636] Lustre: *** cfs_fail_loc=a09, val=0***
[ 5432.278689] Lustre: Skipped 3463 previous similar messages
[ 5732.575238] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5732.577552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5732.577554] Lustre: Skipped 1 previous similar message
[ 5732.588533] Lustre: Skipped 1 previous similar message
[ 5736.678262] LustreError: 9267:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup
[ 5736.681412] LustreError: 9267:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages
[ 5736.772869] Lustre: server umount lustre-MDT0000 complete
[ 5739.009175] LustreError: 23645:0:(ldlm_lockd.c:2549:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1746145633 with bad export cookie 18229127965547296521
[ 5739.015877] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5739.165171] Lustre: server umount lustre-OST0000 complete
[ 5741.450516] LustreError: 9955:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup
[ 5741.452446] LustreError: 9955:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages
[ 5741.521854] Lustre: server umount lustre-OST0001 complete
[ 5744.526463] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_hostid
[ 5747.418854] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local
[ 5751.992741]  vdc: vdc1 vdc9
[ 5756.884026]  vde: vde1 vde9
[ 5762.003137]  vdf: vdf1 vdf9
[ 5768.412809] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local
[ 5772.412131] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000
[ 5772.547476] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[ 5772.589607] Lustre: lustre-MDT0000: new disk, initializing
[ 5772.823243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5772.838708] Lustre: Skipped 1 previous similar message
[ 5772.892087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[ 5774.791835] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5777.924037] Lustre: Modifying parameter general.debug_raw_pointers in log params
[ 5780.775061] Lustre: 15469:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.150@tcp: no remote llog for lustre-sptlrpc, check MGS config
[ 5780.789950] Lustre: lustre-OST0000: new disk, initializing
[ 5780.792599] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[ 5780.795586] Lustre: Skipped 1 previous similar message
[ 5780.847141] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 5782.347641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost
[ 5782.354504] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost]
[ 5782.488998] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400
[ 5783.774837] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5789.392907] Lustre: 16796:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.150@tcp: no remote llog for lustre-sptlrpc, check MGS config
[ 5789.412158] Lustre: lustre-OST0001: new disk, initializing
[ 5789.414953] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[ 5789.462421] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 5791.003812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost
[ 5791.011213] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost]
[ 5791.154388] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400
[ 5792.163213] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5797.583293] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 5799.514708] Lustre: Setting parameter general.lod.*.mdt_hash in log params
[ 5825.955720] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 20:28:40 (1746145720)
[ 5860.982373] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 20:29:14 (1746145754)
[ 5895.807830] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 20:29:49 (1746145789)
[ 5910.624105] Lustre: DEBUG MARKER: rename directory return 255
[ 5935.230649] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 20:30:29 (1746145829)
[ 5954.421016] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 20:30:48 (1746145848)
[ 5984.048690] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 20:31:17 (1746145877)
[ 6029.089361] LustreError: 19430:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-0x0 id:60001 enforced:1 hard:51200 soft:0 granted:51200 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[ 6052.231837] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 20:32:26 (1746145946)
[ 6073.864337] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 20:32:47 (1746145967)
[ 6101.637095] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 20:33:15 (1746145995)
[ 6196.045013] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 20:34:49 (1746146089)
[ 6196.787725] Lustre: DEBUG MARKER: SKIP: sanity-quota test_59 ldiskfs only test
[ 6197.850106] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 20:34:51 (1746146091)
[ 6206.854867] LustreError: 10794:0:(qsd_reint.c:617:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:0 until pending updates are flushed.
[ 6206.859017] LustreError: 10794:0:(qsd_reint.c:617:qqi_reint_delayed()) Skipped 4 previous similar messages
[ 6236.781884] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[ 6237.464560] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 20:35:31 (1746146131)
[ 6256.788417] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[ 6257.889628] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non dir/files should succeed ========================================================== 20:35:51 (1746146151)
[ 6291.663144] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[ 6292.488319] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 20:36:26 (1746146186)
[ 6321.911961] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 20:36:55 (1746146215)
[ 6322.564513] Lustre: DEBUG MARKER: SKIP: sanity-quota test_67 ZFS grants some block space together with inode
[ 6323.396767] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 20:36:57 (1746146217)
[ 6341.125954] LustreError: 17734:0:(mgs_handler.c:848:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[ 6367.829255] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 20:37:41 (1746146261)
[ 6377.049149] LustreError: 20038:0:(qsd_reint.c:617:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:2 until pending updates are flushed.
[ 6377.056944] LustreError: 20038:0:(qsd_reint.c:617:qqi_reint_delayed()) Skipped 5 previous similar messages
[ 6381.294823] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[ 6381.884953] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[ 6481.113722] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 20:39:35 (1746146375)
[ 6510.567801] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 20:40:04 (1746146404)
[ 6531.098377] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 20:40:24 (1746146424)
[ 6531.935020] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71a ZFS grants some block space together with inode
[ 6532.759924] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 20:40:26 (1746146426)
[ 6533.453993] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71b ZFS grants some block space together with inode
[ 6534.282589] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 20:40:28 (1746146428)
[ 6543.760450] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB)
[ 6552.578901] Lustre: DEBUG MARKER: Write...
[ 6553.768587] Lustre: DEBUG MARKER: Write out of block quota ...
[ 6586.650001] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 20:41:20 (1746146480)
[ 6602.889134] Lustre: DEBUG MARKER: set to use default quota
[ 6603.682377] Lustre: DEBUG MARKER: set default quota
[ 6604.511976] Lustre: DEBUG MARKER: get default quota
[ 6606.861621] Lustre: DEBUG MARKER: Test not out of quota
[ 6609.110134] Lustre: DEBUG MARKER: Test out of quota
[ 6614.889736] Lustre: DEBUG MARKER: Increase default quota
[ 6627.564410] Lustre: DEBUG MARKER: Set quota to override default quota
[ 6627.596980] LustreError: 14268:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45063 time:1746751321 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[ 6632.848412] Lustre: DEBUG MARKER: Set to use default quota again
[ 6646.939166] Lustre: DEBUG MARKER: Cleanup
[ 6696.314306] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 20:43:10 (1746146590)
[ 6713.532831] Lustre: DEBUG MARKER: set default quota for qpool1
[ 6714.635211] Lustre: DEBUG MARKER: Write from user that hasn't lqe
[ 6748.039992] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 20:44:01 (1746146641)
[ 6801.832727] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 20:44:55 (1746146695)
[ 6891.969119] Lustre: DEBUG MARKER: Write...
[ 6894.057070] Lustre: DEBUG MARKER: Write out of block quota ...
[ 6998.364423] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 20:48:12 (1746146892)
[ 7027.623614] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 20:48:41 (1746146921)
[ 7031.917482] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 20:48:45 (1746146925)
[ 7032.746169] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78A need >= 2.13.57 and ldiskfs for fallocate
[ 7033.922222] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 20:48:47 (1746146927)
[ 7034.996456] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78a need >= 2.13.57 and ldiskfs for fallocate
[ 7036.114284] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 20:48:49 (1746146929)
[ 7048.148785] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 20:49:02 (1746146942)
[ 7048.830649] Lustre: DEBUG MARKER: SKIP: sanity-quota test_80 ZFS grants some block space together with inode
[ 7049.614276] Lustre: DEBUG MARKER: == sanity-quota test 81: Race qmt_start_pool_recalc with qmt_pool_free ========================================================== 20:49:03 (1746146943)
[ 7060.473492] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[ 7066.549601] LustreError: 22426:0:(qmt_pool.c:1295:qmt_pool_recalc()) cfs_fail_timeout id a07 sleeping for 10000ms
[ 7073.054391] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.50@tcp (stopping)
[ 7073.067943] Lustre: Skipped 1 previous similar message
[ 7073.167044] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 7073.177320] Lustre: Skipped 1 previous similar message
[ 7076.642640] LustreError: 22426:0:(qmt_pool.c:1295:qmt_pool_recalc()) cfs_fail_timeout id a07 awake
[ 7076.753700] LustreError: 22571:0:(obd_class.h:479:obd_check_dev()) Device 17 not setup
[ 7076.756034] LustreError: 22571:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages
[ 7076.842398] Lustre: server umount lustre-MDT0000 complete
[ 7080.939786] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 7081.249436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 7081.353969] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:27 to 0x280000400:65)
[ 7081.366175] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:27 to 0x240000400:65)
[ 7082.989408] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7093.222605] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to  (at 0@lo)
[ 7093.232487] Lustre: Skipped 3 previous similar messages
[ 7113.323826] Lustre: DEBUG MARKER: == sanity-quota test 82: verify more than 8 qids for single operation ========================================================== 20:50:07 (1746147007)
[ 7133.545423] Lustre: DEBUG MARKER: == sanity-quota test 83: Setting default quota shouldn't affect grace time ========================================================== 20:50:27 (1746147027)
[ 7153.749119] Lustre: DEBUG MARKER: == sanity-quota test 84: Reset quota should fix the insane granted quota ========================================================== 20:50:47 (1746147047)
[ 7170.514083] LustreError: 15633:0:(qsd_reint.c:617:qqi_reint_delayed()) lustre-OST0000: Delaying reintegration for qtype:0 until pending updates are flushed.
[ 7170.518793] LustreError: 15633:0:(qsd_reint.c:617:qqi_reint_delayed()) Skipped 2 previous similar messages
[ 7184.970053] Lustre: *** cfs_fail_loc=a08, val=0***
[ 7184.973038] Lustre: Skipped 1337 previous similar messages
[ 7184.977106] Lustre: *** cfs_fail_loc=a08, val=0***
[ 7254.143757] Lustre: DEBUG MARKER: == sanity-quota test 85: do not hung at write with the least_qunit ========================================================== 20:52:27 (1746147147)
[ 7311.119946] Lustre: DEBUG MARKER: == sanity-quota test 86: Pre-acquired quota should be released if quota is over limit ========================================================== 20:53:25 (1746147205)
[ 7441.449829] LustreError: 23279:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1746752135 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[ 7623.669253] LustreError: 23280:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1746752317 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[ 7792.374881] LustreError: 25022:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:1000 enforced:1 hard:2048 soft:2048 granted:65536 time:1746752486 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[ 7858.498427] Lustre: DEBUG MARKER: == sanity-quota test 87: lfs quota -a should print default quota setting ========================================================== 21:02:32 (1746147752)
[ 7866.179108] Lustre: *** cfs_fail_loc=a09, val=0***
[ 7867.184271] Lustre: *** cfs_fail_loc=a09, val=0***
[ 7867.188636] Lustre: Skipped 119 previous similar messages
[ 7869.187815] Lustre: *** cfs_fail_loc=a09, val=0***
[ 7869.189571] Lustre: Skipped 227 previous similar messages
[ 7873.191376] Lustre: *** cfs_fail_loc=a09, val=0***
[ 7873.192865] Lustre: Skipped 518 previous similar messages
[ 7909.442236] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.50@tcp (stopping)
[ 7909.447095] Lustre: Skipped 2 previous similar messages
[ 7909.554733] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 7909.574636] Lustre: Skipped 1 previous similar message
[ 7912.167714] LustreError: 25078:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup
[ 7912.172247] LustreError: 25078:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 7912.271387] Lustre: server umount lustre-MDT0000 complete
[ 7915.447184] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 7915.692080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 7915.754591] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:27 to 0x280000400:97)
[ 7915.758273] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:68 to 0x240000400:97)
[ 7916.887029] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7929.669369] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to  (at 0@lo)
[ 7929.676622] Lustre: Skipped 1 previous similar message
[ 7938.807430] Lustre: DEBUG MARKER: == sanity-quota test 88: Writing over quota should not hang ========================================================== 21:03:52 (1746147832)
[ 7939.334300] Lustre: DEBUG MARKER: SKIP: sanity-quota test_88 require client with >4k pages
[ 7939.987679] Lustre: DEBUG MARKER: == sanity-quota test 89: Show default quota with squash_uid ========================================================== 21:03:54 (1746147834)
[ 7980.389120] Lustre: DEBUG MARKER: == sanity-quota test 90a: lfs quota should work without mount point ========================================================== 21:04:34 (1746147874)
[ 8007.923383] Lustre: DEBUG MARKER: == sanity-quota test 90b: lfs quota should work with multiple mount points ========================================================== 21:05:01 (1746147901)
[ 8016.905633] Lustre: DEBUG MARKER: sanity-quota test_90b: @@@@@@ FAIL: quota info from /mnt/lustre not ' /mnt/lustre 0 11721 0 0 - 375 0 0 -